/Lazy ODB tree
It is created with default parameters on the first activation of the lazylogger utility . This task connects to a defined channel (e.g. Tape, Disk) when started. Multiple instances of the program can run concurrently, connecting to a different channel (see lazylogger above).
The following example shows a /lazy tree with a <channel-name> of "Disk". A sub-tree /lazy/disk has been created, containing sub-trees "Settings" and "Statistics" . A third sub-tree "List" will be created after the first successful operation of the channel. The contents of these sub-trees is discussed below (see Settings, Statistics and List subtrees.)
[local:pol:S]/>ls /lazy/disk Settings Statistics List
Example of the ODB /Lazy/<channel_name>/Settings subtree
Settings DIR
Maintain free space(%) INT 1 4 3m 0 RWD 0
Stay behind INT 1 4 3m 0 RWD -3
Alarm Class STRING 1 32 3m 0 RWD
Running condition STRING 1 128 3m 0 RWD ALWAYS
Data dir STRING 1 256 3m 0 RWD /home/midas/online
Data format STRING 1 8 3m 0 RWD MIDAS
Filename format STRING 1 128 3m 0 RWD run%05d.mid
Backup type STRING 1 8 3m 0 RWD Tape
Execute after rewind STRING 1 64 3m 0 RWD
Path STRING 1 128 3m 0 RWD
Capacity (Bytes) FLOAT 1 4 3m 0 RWD 5e+09
List label STRING 1 128 3m 0 RWD
Execute before writing file STRING 1 64 11h 0 RWD lazy_prewrite.csh
Execute after writing file STRING 1 64 11h 0 RWD rundb_addrun.pl
Modulo.Position STRING 1 8 11h 0 RWD 2.1
Tape Data Append BOOL 1 4 11h 0 RWD y
Explanation of the Keys in the ODB /Lazy/<channel_name>/Settings subtree
The following table explains the meaning of the keys in the Settings subtree. An example of the subtree is shown above.
Above: meaning of keys in the /Lazy/<channel_name>/Settings ODB tree Keys in the ODB tree /Lazy/<channel_name>/Settings
ODB Key
Type
Explanation
Settings
DIR
ODB Settings subtree for a Lazylogger channel.
Maintain free space(%)
INT
As the data Logger (mlogger) runs independently from the Lazylogger, the data logging device (e.g. disk) contains all the recorded data files. Under this condition, Lazylogger can be instructed to "purge" the disk after successful backup of the data onto a "slow device" (e.g. tape). The "Maintain free space(\%)" parameter controls the percentage of disk space required to be maintained as free (for "none" set to 0).
The conditions that must ALL be satisfied before removing a data file are defined as: The data file corresponding to the given run number following the format declared under Settings/Filename format IS PRESENT on the Settings/Data Dir path. The given run number MUST appear anywhere under the Settings/List/ directory of ALL the Lazy channels having the same Settings/Filename format as this channel. The given run number MUST appear anywhere under the Settings/List/ directory of that channel
Stay behind
INT
This parameter defines how many consecutive data files should be kept between the current run and the last lazylogger run.
Example with "Stay behind = -3" : Current acquisition run number 253 -> run00253.mid is being logged by mlogger. Files available on the disk corresponding to runs #248, #249, #250, #251, #252. Lazylogger will start backing up run #250 as soon as the new run 254 starts. "Stay behind = -3" corresponds to 3 files untouched on the disk (#251, #252, #253). If the sign (+/-) of the "Stay behind" parameter is
negative: the lazylogger will always scan the entire disk at the "Data Dir" path, from the oldest to the most recent file present for backup. positive: lazylogger will backup starting from -x behind the current acquisition run number. Run order will be ignored.
Alarm Class
STRING
Specify the Alarm class to be used in case of an alarm being triggered.
Running Condition
STRING
Specify the type of condition for which the lazylogger should be actived. By default lazylogger is ALWAYS running. The choices are
ALWAYS, NEVER, WHILE_NO_ACQ_RUNNING, or a condition based on a single field of the ODB key<=>value Example : In the case of high data rate acquisition it could be necessary to activate the lazylogger only
when the run is either paused or stopped (i.e. WHILE_NO_ACQ_RUNNING)
odbedit> set "Running condition" WHILE_ACQ_NOT_RUNNING
or when some external condition is satisfied such as "low beam intensity".
odbedit> set "Running condition" "/alias/max_rate < 200"
Data dir
STRING
Specifies the data directory path of the data logging device (e.g. disk). If the Key "Data Dir" is present, the contents is taken as the path, otherwise the current directory is used (i.e. the path where the lazylogger client was started).
Data format
STRING
Specifies the Data Format of the data files. Currently supported formats are MIDAS (Note that YBOS is no longer supported - see Sep 2010).
Filename format
STRING
Specifies the file format of the data files on the data logging device (i.e. disk). This should be the same filename format as used by the data logger to save the files.
Backup type
STRING
Specifies the "slow device" backup type. Default Tape. =======
[Data dir] Specify the Data directory path of the data files. By default if the "/Logger/Data Dir" is present, the pointed value is taken otherwise the current directory where lazylogger has been started is used.
[Data format] Specify the Data format of the data files. Currently supported formats are: MIDAS (Note that YBOS format is no longer supported - see Sep 2010 ).
[Filename format] Specify the file format of the data files. Same format as given for the data logger.
[Backup type] Specify the "slow device" backup type. Default Tape. >>>>>>> .r4846 Can be Disk or Ftp.
Execute after rewind
STRING
Specifies an optional script to run after completion of a lazylogger backup set (see below "Capacity (Bytes)").
Path
STRING
Specifies the "slow device" path. Three possible types of Path:
For Tape : /dev/nst0- (UNIX like). For Disk : /data1/myexpt For Ftp : host,port,user,password,directory
Capacity (Bytes)
STRING
Specifies the maximum "slow device" capacity in bytes. When this capacity is reached,the lazylogger will close the backup device and clear the List Label field to prevent further backup. It will also rewind the stream device if possible.
List label
STRING
Specify a label for a set of backed up files to the "slow device". This label is used only internaly by the lazylogger for creating a new array composed of the backed up runs under the /Settings/List subdirectory, until the Capacity value has been reached. Once the backup set is complete, lazylogger will clear the "List label" field and therefore prevent any further backup until a non-empty "List label" is entered again. On the other hand the list label will remain under the /List key to display all run being backed up until the corresponding files have been removed from the disk.
Exec preW file
STRING
Permits a script to be run before the begining of the lazy job. The arguments passed to the scripts are: input file name , output file name, current block number.
Exec postW file
STRING
Permits a script to be run after the completion of the lazy job. The arguments passed to the scripts are: list label, current job number, source path, file name, file size in MB, current block number.
Modulo.Position
STRING
This field is for multiple instances of the lazylogger where each instance works on a sub-set of run number. By specifying the Modulo.Position you're telling the current lazy instance how many instances are simultaneously running (3.) and the position of which this instance is assigned to (.1) As an example for 3 lazyloggers running simultaneously the field assignment should be :
Channel Field Run# Lazy_1 3.0 21, 24, 27, ... Lazy_2 3.1 22, 25, 28, ... Lazy_3 3.2 23, 26, 29, ...
In the example of the /Settings tree above, the Modulo.Position is set to 2.1 , indicating this is instance 1, and two lazyloggers are running.
Tape Data Append
BOOL
If this key is set to "y", the spooling of the Tape device to the End_of_Device (EOD) before starting the lazy job is enabled. This command is valid only for "Backup Type" Tape. If this flag is not enabled the lazy job starts at the current tape position.
The ODB /Lazy/<channel_name>/Statistics subtree
The Statistics ODB subtree contains general information about the status of the current lazylogger channel.
An example of the Statistics subtree for a lazylogger channel is shown below:
[local:pol:S]/>ls /lazy/disk/Statistics/ Backup file 000321.mid File size (Bytes) 0 KBytes copied 0 Total Bytes copied 0 Copy progress (%) 0 Copy Rate (Bytes per s) 0 Backup status (%) 0 Number of Files 173 Current Lazy run 321
The ODB /Lazy/<channel_name>/List subtree
The List ODB subtree will contain arrays of run number associated with the array name backup-set label (see list label) . Any run number appearing in any of the arrays is considered to have been backed up.
An example of the List subtree for a lazylogger channel is shown below. In this case, the list label was set to "archive".
[local:pol:S]/>ls /lazy/disk/List archive
230
231
232
233
234
235
236
237
Setting up the Lazylogger
The Lazylogger requires to be setup before data file can be moved. This setup consists of 4 steps:
Invoke the lazylogger once for setting up the appropriate ODB tree and exit. In this example, the channel is "Tape".
>lazylogger -c Tape
Edit the newly created ODB tree. Set the Settings field(s) to match your requirements. > odbedit -e midas [local:midas:Stopped]/>cd /Lazy/tape/ [local:midas:Stopped]tape>ls [local:midas:Stopped]tape>ls -lr Key name Type #Val Size Last Opn Mode Value
tape DIR
Settings DIR
Maintain free space(%) INT 1 4 3m 0 RWD 0
Stay behind INT 1 4 3m 0 RWD -3
Alarm Class STRING 1 32 3m 0 RWD
Running condition STRING 1 128 3m 0 RWD ALWAYS
Data dir STRING 1 256 3m 0 RWD /home/midas/online
Data format STRING 1 8 3m 0 RWD MIDAS
Filename format STRING 1 128 3m 0 RWD run%05d.mid
Backup type STRING 1 8 3m 0 RWD Tape
Execute after rewind STRING 1 64 3m 0 RWD
Path STRING 1 128 3m 0 RWD
Capacity (Bytes) FLOAT 1 4 3m 0 RWD 5e+09
List label STRING 1 128 3m 0 RWD
Statistics DIR
Backup file STRING 1 128 3m 0 RWD none
File size [Bytes] DOUBLE 1 4 3m 0 RWD 0
KBytes copied DOUBLE 1 4 3m 0 RWD 0
Total Bytes copied DOUBLE 1 4 3m 0 RWD 0
Copy progress [%] DOUBLE 1 4 3m 0 RWD 0
Copy Rate [bytes per s] DOUBLE 1 4 3m 0 RWD 0
Backup status [%] DOUBLE 1 4 3m 0 RWD 0
Number of Files INT 1 4 3m 0 RWD 0
Current Lazy run INT 1 4 3m 0 RWD 0
[local:midas:Stopped]tape>cd Settings/ [local:midas:Stopped]Settings>set "Data dir" /data [local:midas:Stopped]Settings>set "Capacity (Bytes)" 15e9 Start lazylogger in the background >lazylogger -c Tape -D At this point, the lazylogger is running and waiting for the list label key to be defined before starting the copy procedure. The mstat utility will display information regarding the status of the lazylogger.
> odbedit -e midas [local:midas:Stopped]/>cd /Lazy/tape/Settings [local:midas:Stopped]Settings>set "List label" cni-043
Operation of the Lazylogger
For every major operation of the lazylogger a message is sent to the Message buffer and will be appended to the default MIDAS log file (midas.log). These messages are the only mean of finding out What/When/Where/How the lazylogger has operated on a data file. A fragment of the midas.log for the chaos experiment is shown below. In this case, the Maintain free space(%) field was enabled which produces the cleanup of the data files and the entry in the List tree after copy.
Fri Mar 24 14:40:08 2000 [Lazy_Tape] 8351 (rm:16050ms) /scr0/spring2000/run08351.ybs file REMOVED Fri Mar 24 14:40:08 2000 [Lazy_Tape] Tape run#8351 entry REMOVED Fri Mar 24 14:59:55 2000 [Logger] stopping run after having received 1200000 events Fri Mar 24 14:59:56 2000 [CHAOS] Run 8366 stopped Fri Mar 24 14:59:56 2000 [Logger] Run #8366 stopped Fri Mar 24 14:59:57 2000 [SUSI] saving info in run log Fri Mar 24 15:00:07 2000 [Logger] starting new run
Fri Mar 24 15:00:07 2000 [CHAOS] Run 8367 started Fri Mar 24 15:00:07 2000 [Logger] Run #8367 started Fri Mar 24 15:06:59 2000 [Lazy_Tape] cni-043[15] (cp:410.6s) /dev/nst0/run08365.ybs 864.020MB file NEW Fri Mar 24 15:07:35 2000 [Lazy_Tape] 8352 (rm:25854ms) /scr0/spring2000/run08352.ybs file REMOVED Fri Mar 24 15:07:35 2000 [Lazy_Tape] Tape run#8352 entry REMOVED Fri Mar 24 15:27:09 2000 [Lazy_Tape] 8353 (rm:23693ms) /scr0/spring2000/run08353.ybs file REMOVED Fri Mar 24 15:27:09 2000 [Lazy_Tape] Tape run#8353 entry REMOVED Fri Mar 24 15:33:22 2000 [Logger] stopping run after having received 1200000 events Fri Mar 24 15:33:22 2000 [CHAOS] Run 8367 stopped Fri Mar 24 15:33:23 2000 [Logger] Run #8367 stopped Fri Mar 24 15:33:24 2000 [SUSI] saving info in run log Fri Mar 24 15:33:33 2000 [Logger] starting new run Fri Mar 24 15:33:34 2000 [CHAOS] Run 8368 started Fri Mar 24 15:33:34 2000 [Logger] Run #8368 started Fri Mar 24 15:40:18 2000 [Lazy_Tape] cni-043[16] (cp:395.4s) /dev/nst0/run08366.ybs 857.677MB file NEW Fri Mar 24 15:50:15 2000 [Lazy_Tape] 8354 (rm:28867ms) /scr0/spring2000/run08354.ybs file REMOVED Fri Mar 24 15:50:15 2000 [Lazy_Tape] Tape run#8354 entry REMOVED ... Once the lazylogger has started a job on a data file, trying to terminate the application will result in producing a log message informing the user of the actual percentage of the backup completed so far. This message will repeat itself until completion of the backup, and only then the lazylogger application will terminate. If an interruption of the lazylogger is forced (kill...), the state of the backup device is undetermined. Recovery is not possible, and the full backup set has to be redone. In order to do this, you need: To rewind the backup device. Delete the /Lazy/<channel_name>/List/<list label> array. Restart the lazylogger with the -z switch which will "zap" the statistics entries. In order to facilitate the recovery procedure, lazylogger produces an ODB ASCII file of the lazy channel tree after completion of successful operation. This file (Tape_recover.odb) stored in Data dir can be used for ODB as well as lazylogger recovery.