Lazylogger: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
(The midas documentation has moved to https://midas.triumf.ca/docs/)
Tag: Replaced
 
(5 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Pagelinks}}
The midas documentation has moved. See [https://midas.triumf.ca/docs/develop/features/storing_data/data_logging/archiving/ Data logging: archiving (lazylogger)] and [https://midas.triumf.ca/docs/develop/reference/reference/programs/#lazylogger lazylogger] on the Programs and options page for the new documentation about the lazylogger.
== Links ==
<div style="column-count:2;-moz-column-count:2;-webkit-column-count:2">
* [[Data Logger]]
* [[/Lazy ODB tree]]
</div>
 
 
 
== Purpose ==
The <span style="color:darkcyan;">''lazylogger''</span> utility decouples the data acquisition from the data logging mechanism. The need for such an application has been dictated by the slow response time of the old media logging devices, such as tape devices. Delay due to tape mounting, re-tension and re-positioning implies that the data acquisition has to be held until operation completion. By using '''[[mlogger]]''' to log data to disk in a first stage and then using  <span style="color:darkcyan;"> ''lazylogger''</span> to copy or move the stored files, the data acquisition can be kept running without interruption. These devices are no longer supported internally by Midas but can still be managed through scripts. Lazylogger is still much used for getting a copy of the original data to a remote data storage. While there are many ways to do this independently of the acquisition system, the lazylogger provides a simple way to manage this task. Lazylogger features are listed below. 
 
* Multiple  <span style="color:darkcyan;">''lazyloggers''</span> can be running simultaneously on the same computer, each one taking care of a particular channel.
* Each  <span style="color:darkcyan;">''lazylogger''</span> channel will have a dedicated ODB tree (see ODB /Lazy Tree ) containing its own information.
* All the  <span style="color:darkcyan;">''lazylogger''</span> channels will be in the ODB under [[/Lazy ODB tree|/Lazy/<channel-name>/...]].
* Each channel tree is composed of two sub-trees [[/Lazy ODB tree#Explanation of the Keys in the ODB /Lazy/<channel_name>/Settings subtree|Settings]],[[/Lazy ODB tree#The ODB /Lazy/<channel_name>/Statistics subtree|Statistics]] and [[/Lazy ODB tree#The ODB /Lazy/<channel_name>]].
* Dynamic directory destination based on run number or date.
* Script copy support.
* Compression copy.
 
The [[/Lazy ODB tree#Explanation of the Keys in the ODB /Lazy/<channel_name>/Settings subtree|Settings]] and [[/Lazy ODB tree#The ODB /Lazy/<channel_name>/Statistics subtree|Statistics]] trees are self-explanatory, and contain the operational parameters of the channel(s). Two hidden log files for each channel are created and updated ans resides in the main experiment directory
(.<channel_name>.dirlist, .<chanel_name>.donelist), these files reflect the current state of the data copy. Lazylogger support internally data file transfer through FTP repository system. Improvements towards more generic support (including scripting and pipe compression) have been added as well.
 
== Usage ==
 
Usage: lazylogger [-D] [-h <Hostname>] [-e <Experiment>] -c <channel name>
 
== Arguments ==
[-h ] : help.
[-h hostname ]        : see [[Common Parameters to MIDAS Utilities |common parameters]]
[-e experiment_name ] : see [[Common Parameters to MIDAS Utilities |common parameters]]
[-D ]                : start program as a daemon.
[-c channel ]        : logging channel ''<channel-name>''. Specify the lazylogger to activate.
[-z ]                : zap statistics. Clear the statistics tree of all the defined lazylogger channels.
[-t ]                : permit lazylogger to TALK msg.
[-n ]                : No file purging on the source dir.
[-d ]                : Additional debug messages
 
== ODB setting fields ==
 
The settings field covers different operation mode of the lazylogger. Some of which may not be used at the present time.
 
Period                        : Lazylogger check during idle phase (ex: 10) seconds
Maintain free space (%)      : Space on the source side (if exceeded, file will be purged based on status of copy (ex: 0)
Stay behind                  : Keep lazylogger copying runs with run# - "stay behind" (ex: 1)
Alarm Class                  : Assign Alarm class 
Running condition            : running condition of lazylogger (ALWAYS, NEVER, WHILE_ACQ_NOT_RUNNING) (ex: ALWAYS)
Data dir                      : Data source directory (ex: /home/midas/online)
Data format                  : Data format (def: MIDAS)
Filename format              : Data source file name format (ex: run%05d.mid,
Backup type                    STRING  1    8    10h  0  RWD  SFTP
Execute after rewind            STRING  1    64    10h  0  RWD 
Path                            STRING  1    128  10h  0  RWD  sftp://dragon@isdaq01.triumf.ca/home/dragon/mid2asc
Capacity (Bytes)                FLOAT  1    4    10h  0  RWD  5e+09
List label                      STRING  1    128  10h  0  RWD  SFTP
Execute before writing file    STRING  1    64    10h  0  RWD 
Execute after writing file      STRING  1    64    10h  0  RWD 
Modulo.Position                STRING  1    8    10h  0  RWD 
Copy Delay                      INT32  1    4    10h  0  RWD  0
Tape Data Append                BOOL    1    4    10h  0  RWD  y
 
 
Quick man :
The Lazy/Settings tree is composed of the following parameters:
Maintain free space (%)(0): purge source device to maintain free space on the source directory
                      (0) : no purge     
Stay behind  (0)        : If negative number : lazylog runs starting from the OLDEST
                            run file sitting in the 'Dir data' to the current acquisition
                            run minus the 'Stay behind number'
                            If positive number : lazylog starts from the current
                            acquisition run minus 'Stay behind number'
                            Zero : no stay-behind - files are saved as soon as they are closed
Alarm Class              : Specify the Class to be used in case of Tape Full condition
Running condition        : active/deactive lazylogger under given condition i.e:
                          'ALWAYS' (default)    : Independent of the ACQ state ...
                          'NEVER'                : ...
                          'WHILE_ACQ_NOT_RUNNING': ...
                          '/alias/max_rate < 200'    (max_rate is a link)
                          '/equipment/scaler/variables/scal[4] < 23.45'
                          '/equipment/trigger/statistics/events per sec. < 400'
Data dir                  : MIDAS Data Directory (same as "/Logger/Data Dir")
Data format              : Data format (MIDAS)
Filename format          : Run format i.e. "run%05d.mid", or "*.mid.gz" or "*.mid.gz,*.xml"
List label                : Label of destination save_set.
                            Prevent lazylogger to run if not given.
                            Will be reset if maximum capacity reached.
Execute after rewind      : Execute the command <cmd> after rewind complete
                          : args passed are: 'device path' 'channel name' 'list label'
                          : The actual command will look like: <cmd> /dev/nst0 Tape Data_2000
Backup type              : Destination device type (Disk, Tape, Script, SFTP, FTP)
Path                      : Destination path (file.ext, /dev/nst0, ftp...)
                            in case of SFTP type, the 'Path' entry should be:
                            sftp://username@host/path
                            in case of FTP type, the 'Path' entry should be:
                            host, port, user, password, directory, run%05d.mid
Capacity (Bytes)          : Maximum capacity of the destination device.
modulo                    : Enable multiple lazy on same source. Ex: 3ch : 3.0, 3.1, 3.2
tapeAppend                : Enable positioning of the TAPE to EOD before each lazy copy
 
== Setting up the Lazylogger ==
 
The Lazylogger requires to be set up before the data file can be moved. This setup consists of the following steps:
 
: '''1.''' Invoke the lazylogger once for setting up the appropriate ODB ''<channel-name>'' tree and exit. In this example, the ''<channel-name>'' is "Pion".
  >lazylogger -c pion
 
: '''2.''' Edit the newly created ODB tree. Set the [[/Lazy ODB tree#Explanation of the Keys in the ODB /Lazy/''<channel_name>''/Settings subtree|Settings]] field(s) to match your requirements.
::  The following example is shown using [[Odbedit|odbedit]] :
> odbedit -e midas
[local:midas:Stopped]/>cd /Lazy/pion/
[local:midas:Stopped]pion>ls
[local:midas:Stopped]pion>ls -lr
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
pion                          DIR
    Settings                    DIR
        Maintain free space(%)  INT    1    4    3m  0  RWD  0
        Stay behind            INT    1    4    3m  0  RWD  -3
        Alarm Class            STRING  1    32    3m  0  RWD 
        Running condition      STRING  1    128  3m  0  RWD  ALWAYS
        Data dir                STRING  1    256  3m  0  RWD  /home/midas/online
        Data format            STRING  1    8    3m  0  RWD  MIDAS
        Filename format        STRING  1    128  3m  0  RWD  run%05d.mid
        Backup type            STRING  1    8    3m  0  RWD  Pion
        Execute after rewind    STRING  1    64    3m  0  RWD 
        Path                    STRING  1    128  3m  0  RWD 
        Capacity (Bytes)        FLOAT  1    4    3m  0  RWD  5e+09
        List label              STRING  1    128  3m  0  RWD 
    Statistics                  DIR
        Backup file            STRING  1    128  3m  0  RWD  none
        File size [Bytes]      DOUBLE  1    4    3m  0  RWD  0
        KBytes copied          DOUBLE  1    4    3m  0  RWD  0
        Total Bytes copied      DOUBLE  1    4    3m  0  RWD  0
        Copy progress [%]      DOUBLE  1    4    3m  0  RWD  0
        Copy Rate [bytes per s] DOUBLE  1    4    3m  0  RWD  0
        Backup status [%]      DOUBLE  1    4    3m  0  RWD  0
        Number of Files        INT    1    4    3m  0  RWD  0
        Current Lazy run        INT    1    4    3m  0  RWD  0
[local:midas:Stopped]pion>cd Settings/
[local:midas:Stopped]Settings>set "Data dir" /data
[local:midas:Stopped]Settings>set "Capacity (Bytes)" 15e9
 
 
: '''3.''' Start lazylogger in the background
>lazylogger -c Pion -D
 
 
: '''4.''' Define  [[/Lazy ODB tree#List label|List label]] key
:: At this point, the lazylogger is running and waiting for the [[/Lazy ODB tree#List label|List label]] key to be defined before starting the copy procedure.
> odbedit -e midas
[local:midas:Stopped]/>cd /Lazy/pion/Settings
[local:midas:Stopped]Settings>set "List label" cni-043
 
''NOTE:'' The [[Mstat|mstat]] utility will display information regarding the status of the lazylogger.
 
== Operation of the Lazylogger ==
 
For every major operation of the <span style="color:darkcyan;">''lazylogger''</span> a message is sent to the [[Message System|Message buffer]] and will be appended to a default Lazy log file (''lazy.log''). These messages are the only means of finding out What/When/Where/How the lazylogger has operated on a data file. A fragment of the ''lazy.log'' for the iris experiment is shown below. In this case, the [[/Lazy ODB tree#Maintain free space(%)|Maintain free space]] field was enabled which produces the cleanup of the data files and the entry in the  [[/Lazy ODB tree#The ODB /Lazy/<channel_name>/List subtree|List]]  tree after copy. In this particular case, the lazylogger uses a perl script to push the data file to a data storage system using dCache.
 
[local:iris:Stopped]/Lazy>
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Lazy                            DIR
    Dcache                      DIR
        Settings                DIR
            Period              INT    1    4    51h  0  RWD  2
            Maintain free space INT    1    4    51h  0  RWD  15
            Stay behind        INT    1    4    51h  0  RWD  1
            Alarm Class        STRING  1    32    51h  0  RWD 
            Running condition  STRING  1    128  51h  0  RWD  ALWAYS
            Data dir            STRING  1    256  51h  0  RWD  /iris_data0/iris/current
            Data format        STRING  1    8    51h  0  RWD  MIDAS
            Filename format    STRING  1    128  51h  0  RWD  iris_%08d%04d.mid.gz
            Backup type        STRING  1    8    51h  0  RWD  Script
            Execute after rewindSTRING  1    64    51h  0  RWD 
            Path                STRING  1    128  51h  0  RWD  /home/iris/bin/lazy_dcache.perl
            Capacity (Bytes)    FLOAT  1    4    51h  0  RWD  5e+09
            List label          STRING  1    128  51h  0  RWD  Dcache
            Execute before writiSTRING  1    64    51h  0  RWD 
            Execute after writinSTRING  1    64    51h  0  RWD 
            Modulo.Position    STRING  1    8    51h  0  RWD 
            Tape Data Append    BOOL    1    4    51h  0  RWD  y
          Statistics              DIR
            Backup file        STRING  1    128  15h  0  RWDE iris_000035270000.mid.gz
            File size (Bytes)  DOUBLE  1    8    15h  0  RWDE 326345493
            KBytes copied      DOUBLE  1    8    15h  0  RWDE 0
            Total Bytes copied  DOUBLE  1    8    15h  0  RWDE 0
            Copy progress (%)  DOUBLE  1    8    15h  0  RWDE 0
            Copy Rate (Bytes perDOUBLE  1    8    15h  0  RWDE 0
            Backup status (%)  DOUBLE  1    8    15h  0  RWDE 0
            Number of Files    INT    1    4    15h  0  RWDE 247
            Current Lazy run    INT    1    4    15h  0  RWDE 35270000
 
Excerpt of lazy.log:
...
Wed May  6 17:26:00 2015 [Lazy_Dcache] Dcache[30] (cp:1.0s) iris_000032140000.mid.gz 6.146MB  file NEW
Wed May  6 17:26:02 2015 [Lazy_Dcache] Dcache[31] (cp:0.1s) iris_000032150000.mid.gz 22.894MB  file NEW
Wed May  6 17:26:04 2015 [Lazy_Dcache] Dcache[32] (cp:0.1s) iris_000032160000.mid.gz 116.110MB  file NEW
Wed May  6 17:26:07 2015 [Lazy_Dcache] Dcache[33] (cp:0.1s) iris_000032170000.mid.gz 110.228MB  file NEW
Wed May  6 17:26:09 2015 [Lazy_Dcache] Dcache[34] (cp:0.0s) iris_000032180000.mid.gz 101.333MB  file NEW
Wed May  6 17:26:12 2015 [Lazy_Dcache] Dcache[35] (cp:0.1s) iris_000032190000.mid.gz 90.928MB  file NEW
Wed May  6 17:26:14 2015 [Lazy_Dcache] Dcache[36] (cp:0.1s) iris_000032200000.mid.gz 88.461MB  file NEW
Thu May  7 15:47:02 2015 [Lazy_Dcache] Dcache[1] (cp:3.2s) iris_000032210000.mid.gz 87.862MB  file NEW
Thu May  7 15:47:06 2015 [Lazy_Dcache] Dcache[2] (cp:2.2s) iris_000032220000.mid.gz 74.789MB  file NEW
Thu May  7 15:47:10 2015 [Lazy_Dcache] Dcache[3] (cp:1.2s) iris_000032230000.mid.gz 0.204MB  file NEW
Thu May  7 15:47:16 2015 [Lazy_Dcache] Dcache[4] (cp:4.1s) iris_000032240000.mid.gz 105.346MB  file NEW
Thu May  7 15:47:20 2015 [Lazy_Dcache] Dcache[5] (cp:1.2s) iris_000032250000.mid.gz 14.048MB  file NEW
Thu May  7 15:47:23 2015 [Lazy_Dcache] Dcache[6] (cp:1.2s) iris_000032260000.mid.gz 5.011MB  file NEW
Thu May  7 15:47:30 2015 [Lazy_Dcache] Dcache[7] (cp:4.2s) iris_000032270000.mid.gz 94.028MB  file NEW
Thu May  7 15:47:36 2015 [Lazy_Dcache] Dcache[8] (cp:4.2s) iris_000032280000.mid.gz 110.765MB  file NEW
Thu May  7 15:47:42 2015 [Lazy_Dcache] Dcache[9] (cp:3.0s) iris_000032290000.mid.gz 88.982MB  file NEW
Thu May  7 15:47:46 2015 [Lazy_Dcache] Dcache[10] (cp:2.0s) iris_000032300000.mid.gz 51.735MB  file NEW
 
...
Once the lazylogger has started a job on a data file, trying to terminate the application will result in producing a log message informing the user of the actual percentage of the backup completed so far. This message will repeat itself until completion of the backup, and only then the lazylogger application will terminate.
 
[[Category:Utilities]] [[Category:Data Logging]]

Latest revision as of 22:00, 30 October 2026

The midas documentation has moved. See Data logging: archiving (lazylogger) and lazylogger on the Programs and options page for the new documentation about the lazylogger.

Retrieved from "https://daq00.triumf.ca/MidasWiki/index.php?title=Lazylogger&oldid=3739"