Lazylogger: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
No edit summary
(The midas documentation has moved to https://midas.triumf.ca/docs/)
Tag: Replaced
 
(25 intermediate revisions by 2 users not shown)
Line 1: Line 1:
{{Pagelinks}}
The midas documentation has moved. See [https://midas.triumf.ca/docs/develop/features/storing_data/data_logging/archiving/ Data logging: archiving (lazylogger)] and [https://midas.triumf.ca/docs/develop/reference/reference/programs/#lazylogger lazylogger] on the Programs and options page for the new documentation about the lazylogger.
== Links ==
<div style="column-count:2;-moz-column-count:2;-webkit-column-count:2">
* [[Data Logger]]
* [[/Lazy ODB tree]]
</div>
 
 
 
== Purpose ==
The <span style="color:darkcyan;">''lazylogger''</span> utility decouples the data acquisition from the data logging mechanism. The need for such an application has been dictated by the slow response time of some of the media logging devices (i.e. tape devices). Delay due to tape mounting, re-tension and re-positioning implies that the data acquisition has to be held until operation completion. By using '''[[mlogger]]''' to log data to disk in a first stage and then using  <span style="color:darkcyan;">''lazylogger''</span> to copy or move the stored files to the "slow device", the data acquisition can be kept running without interruption.
 
* Multiple  <span style="color:darkcyan;">''lazyloggers''</span> can be running simultaneously on the same computer, each one taking care of a particular channel.
* Each  <span style="color:darkcyan;">''lazylogger''</span> channel will have a dedicated ODB tree (see ODB /Lazy Tree ) containing its own information.
* All the  <span style="color:darkcyan;">''lazylogger''</span> channels will be in the ODB under [[/Lazy ODB tree|/Lazy/<channel-name>/...]].
* Each channel tree is composed of three sub-trees [[/Lazy ODB tree#Explanation of the Keys in the ODB /Lazy/<channel_name>/Settings subtree|Settings]],[[/Lazy ODB tree#The ODB /Lazy/<channel_name>/Statistics subtree|Statistics]] and [[/Lazy ODB tree#The ODB /Lazy/<channel_name>/List subtree|List]].
* Dynamic directory destination based on run number or date.
* Script copy support.
* Compression copy.
 
The [[/Lazy ODB tree#Explanation of the Keys in the ODB /Lazy/<channel_name>/Settings subtree|Settings]] and [[/Lazy ODB tree#The ODB /Lazy/<channel_name>/Statistics subtree|Statistics]] trees are self-explanatory, and contain the operational parameters of the channel(s). The  [[/Lazy ODB tree#The ODB /Lazy/<channel_name>/List subtree|List]] key will have a dynamic list of run numbers which have been successfully dealt with by the lazylogger channel. This list won't exist until the first successful operation of the channel is completed. While the lazylogger was developed specifically for tape devices, it also supports data file transfer to an FTP repository system. Improvements towards more generic support (including scripting and pipe compression) have been added.
 
== Usage ==
 
== Arguments ==
[-h ] : help.
[-h hostname ]        : see [[Common Parameters to MIDAS Utilities |common parameters]]
[-e experiment_name ] : see [[Common Parameters to MIDAS Utilities |common parameters]]
[-D ]                : start program as a daemon.
[-c channel ]        : logging channel ''<channel-name>''. Specify the lazylogger to activate.
[-z ]                : zap statistics. Clear the statistics tree of all the defined lazylogger channels.
 
 
 
 
== Setting up the Lazylogger ==
 
The Lazylogger requires to be set up before the data file can be moved. This setup consists of the following steps:
 
: '''1.''' Invoke the lazylogger once for setting up the appropriate ODB ''<channel-name>'' tree and exit. In this example, the ''<channel-name>'' is "Tape".
  >lazylogger -c Tape
 
: '''2.''' Edit the newly created ODB tree. Set the [[/Lazy ODB tree#Explanation of the Keys in the ODB /Lazy/''<channel_name>''/Settings subtree|Settings]] field(s) to match your requirements.
::  The following example is shown using [[Odbedit|odbedit]] :
> odbedit -e midas
[local:midas:Stopped]/>cd /Lazy/tape/
[local:midas:Stopped]tape>ls
[local:midas:Stopped]tape>ls -lr
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
tape                            DIR
    Settings                    DIR
        Maintain free space(%)  INT    1    4    3m  0  RWD  0
        Stay behind            INT    1    4    3m  0  RWD  -3
        Alarm Class            STRING  1    32    3m  0  RWD 
        Running condition      STRING  1    128  3m  0  RWD  ALWAYS
        Data dir                STRING  1    256  3m  0  RWD  /home/midas/online
        Data format            STRING  1    8    3m  0  RWD  MIDAS
        Filename format        STRING  1    128  3m  0  RWD  run%05d.mid
        Backup type            STRING  1    8    3m  0  RWD  Tape
        Execute after rewind    STRING  1    64    3m  0  RWD 
        Path                    STRING  1    128  3m  0  RWD 
        Capacity (Bytes)        FLOAT  1    4    3m  0  RWD  5e+09
        List label              STRING  1    128  3m  0  RWD 
    Statistics                  DIR
        Backup file            STRING  1    128  3m  0  RWD  none
        File size [Bytes]      DOUBLE  1    4    3m  0  RWD  0
        KBytes copied          DOUBLE  1    4    3m  0  RWD  0
        Total Bytes copied      DOUBLE  1    4    3m  0  RWD  0
        Copy progress [%]      DOUBLE  1    4    3m  0  RWD  0
        Copy Rate [bytes per s] DOUBLE  1    4    3m  0  RWD  0
        Backup status [%]      DOUBLE  1    4    3m  0  RWD  0
        Number of Files        INT    1    4    3m  0  RWD  0
        Current Lazy run        INT    1    4    3m  0  RWD  0
[local:midas:Stopped]tape>cd Settings/
[local:midas:Stopped]Settings>set "Data dir" /data
[local:midas:Stopped]Settings>set "Capacity (Bytes)" 15e9
 
 
: '''3.''' Start lazylogger in the background
>lazylogger -c Tape -D
 
 
: '''4.''' Define  [[/Lazy ODB tree#List label|List label]] key
:: At this point, the lazylogger is running and waiting for the [[/Lazy ODB tree#List label|List label]] key to be defined before starting the copy procedure.
> odbedit -e midas
[local:midas:Stopped]/>cd /Lazy/tape/Settings
[local:midas:Stopped]Settings>set "List label" cni-043
 
''NOTE:'' The [[Mstat|mstat]] utility will display information regarding the status of the lazylogger.
 
== Operation of the Lazylogger ==
 
For every major operation of the <span style="color:darkcyan;">''lazylogger''</span> a message is sent to the [[Message System|Message buffer]] and will be appended to the default MIDAS log file (''midas.log'') See [[Keys in the ODB /Logger tree#/Logger/Message File|Message file]]  . These messages are the only means of finding out What/When/Where/How the lazylogger has operated on a data file. A fragment of the ''midas.log'' for the chaos experiment is shown below. In this case, the [[/Lazy ODB tree#Maintain free space(%)|Maintain free space]] field was enabled which produces the cleanup of the data files and the entry in the  [[/Lazy ODB tree#The ODB /Lazy/<channel_name>/List subtree|List]]  tree after copy.
 
Fri Mar 24 14:40:08 2000 [Lazy_Tape] 8351 (rm:16050ms) /scr0/spring2000/run08351.ybs file REMOVED
Fri Mar 24 14:40:08 2000 [Lazy_Tape] Tape run#8351 entry REMOVED
Fri Mar 24 14:59:55 2000 [Logger] stopping run after having received 1200000 events
Fri Mar 24 14:59:56 2000 [CHAOS] Run 8366 stopped
Fri Mar 24 14:59:56 2000 [Logger] Run #8366 stopped
Fri Mar 24 14:59:57 2000 [SUSI] saving info in run log
Fri Mar 24 15:00:07 2000 [Logger] starting new run
Fri Mar 24 15:00:07 2000 [CHAOS] Run 8367 started
Fri Mar 24 15:00:07 2000 [Logger] Run #8367 started
Fri Mar 24 15:06:59 2000 [Lazy_Tape] cni-043[15] (cp:410.6s) /dev/nst0/run08365.ybs 864.020MB file NEW
Fri Mar 24 15:07:35 2000 [Lazy_Tape] 8352 (rm:25854ms) /scr0/spring2000/run08352.ybs file REMOVED
Fri Mar 24 15:07:35 2000 [Lazy_Tape] Tape run#8352 entry REMOVED
Fri Mar 24 15:27:09 2000 [Lazy_Tape] 8353 (rm:23693ms) /scr0/spring2000/run08353.ybs file REMOVED
Fri Mar 24 15:27:09 2000 [Lazy_Tape] Tape run#8353 entry REMOVED
Fri Mar 24 15:33:22 2000 [Logger] stopping run after having received 1200000 events
Fri Mar 24 15:33:22 2000 [CHAOS] Run 8367 stopped
Fri Mar 24 15:33:23 2000 [Logger] Run #8367 stopped
Fri Mar 24 15:33:24 2000 [SUSI] saving info in run log
Fri Mar 24 15:33:33 2000 [Logger] starting new run
Fri Mar 24 15:33:34 2000 [CHAOS] Run 8368 started
Fri Mar 24 15:33:34 2000 [Logger] Run #8368 started
Fri Mar 24 15:40:18 2000 [Lazy_Tape] cni-043[16] (cp:395.4s) /dev/nst0/run08366.ybs 857.677MB file NEW
Fri Mar 24 15:50:15 2000 [Lazy_Tape] 8354 (rm:28867ms) /scr0/spring2000/run08354.ybs file REMOVED
Fri Mar 24 15:50:15 2000 [Lazy_Tape] Tape run#8354 entry REMOVED
...
Once the lazylogger has started a job on a data file, trying to terminate the application will result in producing a log message informing the user of the actual percentage of the backup completed so far. This message will repeat itself until completion of the backup, and only then the lazylogger application will terminate.
If an interruption of the lazylogger is forced (kill...), the state of the backup device is undetermined. Recovery is not possible, and the full backup set has to be redone. In order to do this, you need:
#To rewind the backup device.
#Delete the /Lazy/<channel_name>/List/<list label> array.
#Restart the lazylogger with the -z switch which will "zap" the statistics entries.
In order to facilitate the recovery procedure, lazylogger produces an ODB ASCII file of the lazy channel tree after completion of successful operation. This file (''Tape_recover.odb'') stored in [[/Lazy ODB tree#Data dir|Data dir]] can be used for ODB as well as lazylogger recovery.
 
[[Category:Utilities]] [[Category:Data Logging]]

Latest revision as of 22:00, 30 October 2026

The midas documentation has moved. See Data logging: archiving (lazylogger) and lazylogger on the Programs and options page for the new documentation about the lazylogger.

Retrieved from "https://daq00.triumf.ca/MidasWiki/index.php?title=Lazylogger&oldid=3739"