Lazylogger
Links
Purpose
The lazylogger utility decouples the data acquisition from the data logging mechanism. The need for such an application has been dictated by the slow response time of the old media logging devices, such as tape devices. Delay due to tape mounting, re-tension and re-positioning implies that the data acquisition has to be held until operation completion. By using mlogger to log data to disk in a first stage and then using lazylogger to copy or move the stored files, the data acquisition can be kept running without interruption. These devices are no longer supported internally by Midas but can still be managed through scripts. Lazylogger is still much used for getting a copy of the original data to a remote data storage. While there are many ways to do this independently of the acquisition system, the lazylogger provides a simple way to manage this task. Lazylogger features are listed below.
- Multiple lazyloggers can be running simultaneously on the same computer, each one taking care of a particular channel.
- Each lazylogger channel will have a dedicated ODB tree (see ODB /Lazy Tree ) containing its own information.
- All the lazylogger channels will be in the ODB under /Lazy/<channel-name>/....
- Each channel tree is composed of two sub-trees Settings,Statistics and /Lazy ODB tree#The ODB /Lazy/<channel_name>.
- Dynamic directory destination based on run number or date.
- Script copy support.
- Compression copy.
The Settings and Statistics trees are self-explanatory, and contain the operational parameters of the channel(s). Two hidden log files for each channel are created and updated ans resides in the main experiment directory (.<channel_name>.dirlist, .<chanel_name>.donelist), these files reflect the current state of the data copy. Lazylogger support internally data file transfer through FTP repository system. Improvements towards more generic support (including scripting and pipe compression) have been added as well.
Usage
Usage: lazylogger [-D] [-h <Hostname>] [-e <Experiment>] -c <channel name>
Arguments
[-h ] : help. [-h hostname ] : see common parameters [-e experiment_name ] : see common parameters [-D ] : start program as a daemon. [-c channel ] : logging channel <channel-name>. Specify the lazylogger to activate. [-z ] : zap statistics. Clear the statistics tree of all the defined lazylogger channels. [-t ] : permit lazylogger to TALK msg. [-n ] : No file purging on the source dir. [-d ] : Additional debug messages
ODB setting fields
The settings field covers different operation mode of the lazylogger. Some of which may not be used at the present time.
Period : Lazylogger check during idle phase (ex: 10) seconds
Maintain free space (%) : Space on the source side (if exceeded, file will be purged based on status of copy (ex: 0)
Stay behind : Keep lazylogger copying runs with run# - "stay behind" (ex: 1)
Alarm Class : Assign Alarm class
Running condition : running condition of lazylogger (ALWAYS, NEVER, WHILE_ACQ_NOT_RUNNING) (ex: ALWAYS)
Data dir : Data source directory (ex: /home/midas/online)
Data format : Data format (def: MIDAS)
Filename format : Data source file name format (ex: run%05d.mid)
In case of script, the file can be of any type (ex: iris_%08d%04d.mid.gz, run%05dsub%03d.mid.lz4)
Backup type : Copy method (Disk, Tape, Script, SFTP, FTP)
Execute after rewind : Script to be performed by lazylogger side in "backup Type=TAPE"
Path : Destination path (ex:
DISK : /data/dragon/Summer-2019/
SCRIPT: /home/dsproto/online/dsproto_daq/lazy_to_analyzer.sh
SCRIPT: /home/agdaq/online/bin/lazy_eos.perl
SFTP : sftp://user@host.domain.ca/home/user/data
public ssh-key is to be setup on the destination host.
Capacity (Bytes) : Maximum capacity of the destination device, mainly for "TAPE" (ex: 5e+09)
List label : Lazylogger channel name (no default, to be filled by user to activate lazylogger) (ex: Mybackup)
Execute before writing file : Script to run before each lazylogger copy operation
Execute after writing file : Script to run after each completion of lazylogger copy operation
Modulo.Position : Enable multiple lazy on same source. Ex: 3ch : 3.0, 3.1, 3.2
Copy Delay : Run transition handling during script operation (def: 0)
Tape Data Append : Enable positioning of the "TAPE" to EOD before each lazy copy (def: y)
Setting up the Lazylogger
The Lazylogger requires to be set up before the data file can be moved. This setup consists of the following steps:
- 1. Invoke the lazylogger once for setting up the appropriate ODB <channel-name> tree and exit. In this example, the <channel-name> is "Pion".
>lazylogger -c pion
- 2. Edit the newly created ODB tree. Set the Settings field(s) to match your requirements.
- The following example is shown using odbedit :
> odbedit -e midas
[local:midas:Stopped]/>cd /Lazy/pion/
[local:midas:Stopped]pion>ls
[local:midas:Stopped]pion>ls -lr
Key name Type #Val Size Last Opn Mode Value
---------------------------------------------------------------------------
pion DIR
Settings DIR
Maintain free space(%) INT 1 4 3m 0 RWD 0
Stay behind INT 1 4 3m 0 RWD -3
Alarm Class STRING 1 32 3m 0 RWD
Running condition STRING 1 128 3m 0 RWD ALWAYS
Data dir STRING 1 256 3m 0 RWD /home/midas/online
Data format STRING 1 8 3m 0 RWD MIDAS
Filename format STRING 1 128 3m 0 RWD run%05d.mid
Backup type STRING 1 8 3m 0 RWD Pion
Execute after rewind STRING 1 64 3m 0 RWD
Path STRING 1 128 3m 0 RWD
Capacity (Bytes) FLOAT 1 4 3m 0 RWD 5e+09
List label STRING 1 128 3m 0 RWD
Statistics DIR
Backup file STRING 1 128 3m 0 RWD none
File size [Bytes] DOUBLE 1 4 3m 0 RWD 0
KBytes copied DOUBLE 1 4 3m 0 RWD 0
Total Bytes copied DOUBLE 1 4 3m 0 RWD 0
Copy progress [%] DOUBLE 1 4 3m 0 RWD 0
Copy Rate [bytes per s] DOUBLE 1 4 3m 0 RWD 0
Backup status [%] DOUBLE 1 4 3m 0 RWD 0
Number of Files INT 1 4 3m 0 RWD 0
Current Lazy run INT 1 4 3m 0 RWD 0
[local:midas:Stopped]pion>cd Settings/
[local:midas:Stopped]Settings>set "Data dir" /data
[local:midas:Stopped]Settings>set "Capacity (Bytes)" 15e9
- 3. Start lazylogger in the background
>lazylogger -c Pion -D
- 4. Define List label key
- At this point, the lazylogger is running and waiting for the List label key to be defined before starting the copy procedure.
> odbedit -e midas [local:midas:Stopped]/>cd /Lazy/pion/Settings [local:midas:Stopped]Settings>set "List label" cni-043
NOTE: The mstat utility will display information regarding the status of the lazylogger.
Using a script
If you set a "Backup type" of "Script", then your custom script (specified as the "Path") will be called for each file that gets written. This script could implement custom logic for moving or renaming files, or do some quick processing of the file. The script can be written in any language.
The script will be called with one argument - the full path to the midas file to process. It should return 0 if it processed the file successfully, and >0 if it failed. lazylogger will automatically retry files that failed. Anything written by the script to stdout will appear in the midas message log.
Note that only 1 instance of the script is ever run at once - if you are writing new files every 60s, but your script takes 90s to execute, you will get a backlog of data and should consider using a proper queue system to implement parallel processing, rather than using the lazylogger.
Operation of the Lazylogger
For every major operation of the lazylogger a message is sent to the Message buffer and will be appended to a default Lazy log file (lazy.log). These messages are the only means of finding out What/When/Where/How the lazylogger has operated on a data file. A fragment of the lazy.log for the iris experiment is shown below. In this case, the Maintain free space field was enabled which produces the cleanup of the data files and the entry in the List tree after copy. In this particular case, the lazylogger uses a perl script to push the data file to a data storage system using dCache.
[local:iris:Stopped]/Lazy>
Key name Type #Val Size Last Opn Mode Value
---------------------------------------------------------------------------
Lazy DIR
Dcache DIR
Settings DIR
Period INT 1 4 51h 0 RWD 2
Maintain free space INT 1 4 51h 0 RWD 15
Stay behind INT 1 4 51h 0 RWD 1
Alarm Class STRING 1 32 51h 0 RWD
Running condition STRING 1 128 51h 0 RWD ALWAYS
Data dir STRING 1 256 51h 0 RWD /iris_data0/iris/current
Data format STRING 1 8 51h 0 RWD MIDAS
Filename format STRING 1 128 51h 0 RWD iris_%08d%04d.mid.gz
Backup type STRING 1 8 51h 0 RWD Script
Execute after rewindSTRING 1 64 51h 0 RWD
Path STRING 1 128 51h 0 RWD /home/iris/bin/lazy_dcache.perl
Capacity (Bytes) FLOAT 1 4 51h 0 RWD 5e+09
List label STRING 1 128 51h 0 RWD Dcache
Execute before writiSTRING 1 64 51h 0 RWD
Execute after writinSTRING 1 64 51h 0 RWD
Modulo.Position STRING 1 8 51h 0 RWD
Tape Data Append BOOL 1 4 51h 0 RWD y
Statistics DIR
Backup file STRING 1 128 15h 0 RWDE iris_000035270000.mid.gz
File size (Bytes) DOUBLE 1 8 15h 0 RWDE 326345493
KBytes copied DOUBLE 1 8 15h 0 RWDE 0
Total Bytes copied DOUBLE 1 8 15h 0 RWDE 0
Copy progress (%) DOUBLE 1 8 15h 0 RWDE 0
Copy Rate (Bytes perDOUBLE 1 8 15h 0 RWDE 0
Backup status (%) DOUBLE 1 8 15h 0 RWDE 0
Number of Files INT 1 4 15h 0 RWDE 247
Current Lazy run INT 1 4 15h 0 RWDE 35270000
Excerpt of lazy.log: ... Wed May 6 17:26:00 2015 [Lazy_Dcache] Dcache[30] (cp:1.0s) iris_000032140000.mid.gz 6.146MB file NEW Wed May 6 17:26:02 2015 [Lazy_Dcache] Dcache[31] (cp:0.1s) iris_000032150000.mid.gz 22.894MB file NEW Wed May 6 17:26:04 2015 [Lazy_Dcache] Dcache[32] (cp:0.1s) iris_000032160000.mid.gz 116.110MB file NEW Wed May 6 17:26:07 2015 [Lazy_Dcache] Dcache[33] (cp:0.1s) iris_000032170000.mid.gz 110.228MB file NEW Wed May 6 17:26:09 2015 [Lazy_Dcache] Dcache[34] (cp:0.0s) iris_000032180000.mid.gz 101.333MB file NEW Wed May 6 17:26:12 2015 [Lazy_Dcache] Dcache[35] (cp:0.1s) iris_000032190000.mid.gz 90.928MB file NEW Wed May 6 17:26:14 2015 [Lazy_Dcache] Dcache[36] (cp:0.1s) iris_000032200000.mid.gz 88.461MB file NEW Thu May 7 15:47:02 2015 [Lazy_Dcache] Dcache[1] (cp:3.2s) iris_000032210000.mid.gz 87.862MB file NEW Thu May 7 15:47:06 2015 [Lazy_Dcache] Dcache[2] (cp:2.2s) iris_000032220000.mid.gz 74.789MB file NEW Thu May 7 15:47:10 2015 [Lazy_Dcache] Dcache[3] (cp:1.2s) iris_000032230000.mid.gz 0.204MB file NEW Thu May 7 15:47:16 2015 [Lazy_Dcache] Dcache[4] (cp:4.1s) iris_000032240000.mid.gz 105.346MB file NEW Thu May 7 15:47:20 2015 [Lazy_Dcache] Dcache[5] (cp:1.2s) iris_000032250000.mid.gz 14.048MB file NEW Thu May 7 15:47:23 2015 [Lazy_Dcache] Dcache[6] (cp:1.2s) iris_000032260000.mid.gz 5.011MB file NEW Thu May 7 15:47:30 2015 [Lazy_Dcache] Dcache[7] (cp:4.2s) iris_000032270000.mid.gz 94.028MB file NEW Thu May 7 15:47:36 2015 [Lazy_Dcache] Dcache[8] (cp:4.2s) iris_000032280000.mid.gz 110.765MB file NEW Thu May 7 15:47:42 2015 [Lazy_Dcache] Dcache[9] (cp:3.0s) iris_000032290000.mid.gz 88.982MB file NEW Thu May 7 15:47:46 2015 [Lazy_Dcache] Dcache[10] (cp:2.0s) iris_000032300000.mid.gz 51.735MB file NEW
... Once the lazylogger has started a job on a data file, trying to terminate the application will result in producing a log message informing the user of the actual percentage of the backup completed so far. This message will repeat itself until completion of the backup, and only then the lazylogger application will terminate.