/Alarms ODB tree: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
No edit summary
mNo edit summary
 
(12 intermediate revisions by the same user not shown)
Line 1: Line 1:
<div style="column-count:4;-moz-column-count:4;-webkit-column-count:4">
{{Pagelinks}}
* [[Midas_documentation|Midas Documentation]]
 
* [[Feature_listing|Feature Listing]]
= Links =
* [[Application_listing|Application Listing]]
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[Online_Database|Online Database]]
* [[Alarms Page]]
* [[Alarm System]]
* [[Mhttpd|mhttpd MIDAS web server]]
* [[odbedit]]
</div>
</div>
= Purpose =
The ODB /Alarms tree contains user and system information related to alarms (see [[Alarm System]]). The information from this tree is displayed in the [[Alarms Page]].


= Creating the /Alarms tree =
When the ODB is created, the /Alarms tree is automatically created, with
* two Alarms with  [[#<alarm-name> subtree|<alarm name>]] "Demo ODB" and "Demo periodic" and
* two Classes with [[#<class-name> subtree|<class name>]] "Alarm" and "Warning".


== Purpose ==
These are shown in the [[#Example of an /Alarms tree|example below]].
The ODB /Alarms tree contains user and system information related to alarms.
 
== Creating the /Alarms tree ==
When the ODB is created, two Classes of alarm are created:
* Alarm
** Demo ODB
** Demo Periodic
* Warning


by default, the alarm system is NOT active
The user may create other alarms and classes by copying an existing <alarm-name> or <alarm-class> subtree and editing (via [[Alarms Page]] or [[odbedit]]) as required.
Currently, the overall alarm is checked once every minute. Once the alarm has been triggered, the message associated with the alarm can be repeated at a different rate.  


By default, the [[Alarm System]] is NOT active. When it is active,
the overall alarm is checked once every minute. Once the alarm has been triggered, the message associated with the alarm can be repeated at a different rate.




== Example of an /Alarms tree ==
= Example =
 
This example (using [[odbedit]]) shows the  default <span style="color: purple; font-style:italic;">/Alarms</span> tree. See also [[Alarms Page]].
This /Alarms tree is shown using [[odbedit]] (see also mhttpd Alarm page).


  $ odbedit
  $ odbedit
Line 61: Line 62:
             Alarm Class        STRING  1    32    >99d 0  RWD  Warning
             Alarm Class        STRING  1    32    >99d 0  RWD  Warning
             Alarm Message      STRING  1    80    >99d 0  RWD  Please do your shift checks
             Alarm Message      STRING  1    80    >99d 0  RWD  Please do your shift checks
      fePOL                  DIR
            Active              BOOL    1    4    19s  0  RWD  y
            Triggered          INT    1    4    19s  0  RWD  205
            Type                INT    1    4    3s  0  RWD  2
            Check interval      INT    1    4    19s  0  RWD  60
            Checked last        DWORD  1    4    19s  0  RWD  1259196026
            Time triggered firstSTRING  1    32    19s  0  RWD  Wed Nov 25 12:59:33 2009
            Time triggered last STRING  1    32    19s  0  RWD  Wed Nov 25 16:40:26 2009
            Condition          STRING  1    256  3s  0  RWD  Program not running
            Alarm Class        STRING  1    32    19s  0  RWD  Caution
            Alarm Message      STRING  1    80    19s  0  RWD  Program fePOL is not running
  Classes                      DIR
        Alarm                  DIR
            Write system messageBOOL    1    4    27h  0  RWD  y
            Write Elog message  BOOL    1    4    27h  0  RWD  n
            System message interINT    1    4    27h  0  RWD  60
            System message last DWORD  1    4    27h  0  RWD  0
            Execute command    STRING  1    256  27h  0  RWD
            Execute interval    INT    1    4    27h  0  RWD  0
            Execute last        DWORD  1    4    27h  0  RWD  0
            Stop run            BOOL    1    4    27h  0  RWD  n
            Display BGColor    STRING  1    32    27h  0  RWD  red
            Display FGColor    STRING  1    32    27h  0  RWD  black
        Warning                DIR
            Write system messageBOOL    1    4    >99d 0  RWD  y
            Write Elog message  BOOL    1    4    >99d 0  RWD  n
            System message interINT    1    4    >99d 0  RWD  60
            System message last DWORD  1    4    >99d 0  RWD  0
            Execute command    STRING  1    256  >99d 0  RWD
            Execute interval    INT    1    4    >99d 0  RWD  0
            Execute last        DWORD  1    4    >99d 0  RWD  0
            Stop run            BOOL    1    4    >99d 0  RWD  n
            Display BGColor    STRING  1    32    >99d 0  RWD  red
            Display FGColor    STRING  1    32    >99d 0  RWD  black


== /Alarms tree structure ==
The Alarms structure is split into 2 sections:
*"Alarms" which define the condition to be tested. The user can create as many Alarms as desired, but each must be one of the four defined Alarm Types .
*"Classes" which define the action to be taken when the alarm occurs. Two Classes (Alarm and Warning) are defined by default. The user can add more Classes as desired.
'''<div id="Alarm Type"></div>'''
The four available Alarm Types are shown in the following table. They are defined in midas.h.
{|  style="text-align: left; width: 50%; background-color: rgb(255, 255, 255);" border="3" cellpadding="2" cellspacing="2"
|-
| colspan="2" rowspan="1" style="vertical-align: top; background-color: mediumslateblue; color:white; text-align:center; font-weight: bold;" | Alarm Type
| colspan="1" rowspan="1" style="vertical-align: top; background-color: mediumslateblue; color:white; text-align:center; font-weight: bold;" | INT value
| colspan="1" rowspan="1" style="vertical-align: top; background-color: mediumslateblue; color:white; text-align:center; font-weight: bold;" | Explanation
|- 
| style=" font-weight: bold;"  |Internal
| AT_INTERNAL
| 1
| Trigger on internal (program) alarm setting through the use of the al_...() functions.
|- 
| style="font-weight: bold;"|Program
| AT_PROGRAM
| 2
| Triggered on condition of the state of the defined task.
|- 
|  style=" font-weight: bold;"  |Evaluated
| AT_EVALUATED
| 3
| Triggered by ODB value on given arithmetical condition.
|- 
| style="font-weight: bold;" |Periodic
| AT_PERIODIC
| 4
| Triggered by timeout condition defined in the alarm setting.
|}
In order to make the system flexible, each alarm class may perform different actions when an alarm is given. For example, it may
* write a system message,
* write to the elog,
* stop the run or
* spawn a detached script listed in the ODB variable [[/Programs/Classes/Execute command]]. This feature is used when an Alarm triggers Email or SMS alerts .
Evaluated Alarm conditions
The alarm condition for evaluated alarms is entered into the ODB key /Alarms/Alarms/<alarm_name>/Condition where <alarm_name> is the name of the alarm. See condition key.
The condition may be simply a comparison between any ODB variable and a threshold parameter, e.g.
/Runinfo/Run number > 100
or it may be an evaluated condition. One can write conditions like
  /Equipment/HV/Variables/Input[*] < 100
or


  /Equipment/HV/Variables/Input[2-3] < 100
= /Alarms tree structure =
to check all values from an array or a certain range. If one array element fulfills the alarm condition, the alarm is triggerrd. In addition, bit-wise alarm conditions are possible, e.g.
The <span  style="color: purple;">/Alarms</span> ODB tree is split into 2 subtrees:
* [[#Alarms subtree|Alarms subtree]] containing as many alarms (i.e. <span  style="color: purple;"><alarm-name></span> subtrees) as the user desires. Each alarm must be one of the four defined [[Alarm System#Alarm Types|Alarm Types]]. These also define the conditions that trigger the alarm.  
* [[#Classes subtree|Classes subtree]] containing as many classes (i.e. <span  style="color: purple;"><class-name></span> subtrees)  as the user desires. Each class defines the action to be taken when the alarm occurs. Two Classes (Alarm and Warning) are defined by default (see [[#Example|Example]]).


  /Equipment/Environment/Variables/Input[0] & 8
See also [[Alarm System]].
The alarm is triggered if bit #2 is set in Input[0].


<br><br>


= Keys in the  <span  style="color: purple;">''/Alarms''</span> ODB tree  =


 
==  <span  style="color: purple;">''Alarm system active''</span>  ==
== Keys in the  <span  style="color: purple;">''/Alarms''</span> ODB tree  ==
 
===  <span  style="color: purple;">''Alarm system active''</span>  ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Type:''' BOOL
Line 155: Line 123:
<br>
<br>


===  <span  style="color: purple;">''Alarms''</span> subtree  ===
==  <span  style="color: purple;">''Alarms''</span> subtree  ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' DIR
* '''Type:''' DIR
Line 170: Line 138:




====  <span  style="color: purple;">''<alarm-name>''</span> subtree  ====
===  <span  style="color: purple;">''<alarm-name>''</span> subtree  ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' DIR
* '''Type:''' DIR
Line 176: Line 144:




This subtree in the [[#Alarms subtree|/Alarms/alarms subtree]] specifies the ''name'' of one of the defined alarms. This subtree will be repeated with a different name for each defined alarm.
This subtree in the [[#Alarms subtree|/Alarms/alarms subtree]] specifies the ''name'' of one of the defined alarms, where ''name'' is substituted for  <alarm-name>. This subtree will be repeated with a different name for each defined alarm.
In the [[#Example|example]] two ''<alarm-name>'' subtrees are shown, called "''demo odb''" and  "''demo periodic''"
In the [[#Example|example]] two ''<alarm-name>'' subtrees are shown, called "''demo odb''" and  "''demo periodic''"


Line 185: Line 153:




=====  <span  style="color: purple;">''Active''</span> =====
====  <span  style="color: purple;">''Active''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Type:''' BOOL
Line 201: Line 169:




=====  <span  style="color: purple;">''Triggered''</span> =====
===  <span  style="color: purple;">''Triggered''</span> ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Type:''' INT
Line 217: Line 185:




=====  <span  style="color: purple;">''Type''</span> =====
====  <span  style="color: purple;">''Type''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Type:''' INT
Line 232: Line 200:




=====  <span  style="color: purple;">''Check interval''</span> =====
====  <span  style="color: purple;">''Check interval''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Type:''' INT
Line 249: Line 217:




=====  <span  style="color: purple;">''Checked last''</span> =====
====  <span  style="color: purple;">''Checked last''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' DWORD
* '''Type:''' DWORD
Line 266: Line 234:




=====  <span  style="color: purple;">''Time triggered first''</span> =====
====  <span  style="color: purple;">''Time triggered first''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Type:''' STRING
Line 282: Line 250:




=====  <span  style="color: purple;">''Time triggered last''</span> =====
====  <span  style="color: purple;">''Time triggered last''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Type:''' STRING
Line 299: Line 267:




=====  <span  style="color: purple;">''Condition''</span> =====
====  <span  style="color: purple;">''Condition''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Type:''' STRING
Line 307: Line 275:


This key in the [[#<alarm-name> subtree|<alarm-name> subtree]]  contains
This key in the [[#<alarm-name> subtree|<alarm-name> subtree]]  contains
the condition on which alarm should trigger.
the condition on which alarm should trigger for [[#/Alarms tree structure|evaluated alarms]]. See [[#Evaluated Alarm conditions]].




Line 316: Line 284:




=====  <span  style="color: purple;">''Alarm class''</span> =====
====  <span  style="color: purple;">''Alarm class''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Type:''' STRING
Line 323: Line 291:




This key in the [[#<alarm-name> subtree|<alarm-name> subtree]] is set to one of the existing Alarm classes, e.g. "Alarm","Warning".
This key in the [[#<alarm-name> subtree|<alarm-name> subtree]] is set to one of the existing Alarm classes, e.g. the defaults "Alarm","Warning" or a user-defined class, e.g. "Caution".
 
The Alarm class must be defined in the [[#<class-name> subtree|Classes subtree]].


<br>
<br>
Line 331: Line 299:




=====  <span  style="color: purple;">''Alarm message''</span> =====
====  <span  style="color: purple;">''Alarm message''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Type:''' STRING
Line 342: Line 310:




<br>
---------
<br>


Classes


==  <span  style="color: purple;">''Classes''</span> subtree ==
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' DIR
</div>


Sub-tree in the /Alarms ODB tree defining each individual alarm class.
Each class defines the individual action to be performed by a predefined and requested alarm.


DIR
<br>
---------
<br>


Sub-tree defining each individual action to be performed by a pre-defined and requested alarm.




===  <span  style="color: purple;">''<class-name>''</span> subtree ===
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' DIR
</div>


Warning
This subtree in the [[#Classes subtree|Classes subtree]] defines one of the alarm classes of name ''name'', where ''name'' is substituted for <class-name>. This subtree will be repeated with a different name for each defined class. In the [[#Example of an /Alarms tree|example above]], two <class-name> subtrees are shown, named "Alarm" and "Warning". 




DIR
<br>
---------
<br>


Name of one of the defined classes




====  <span  style="color: purple;">''Write System Message''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Default:'''  "y"
</div>




Write System Message
If this key in the [[#<class-name> subtree|<class-name> subtree]] is set to "y", a message will be sent to the [[System log]] when an alarm with this [[#Alarm class|alarm class]] is triggered.


BOOL
<br>
---------
<br>


If set to "y" a message will be sent to the System log when alarm is triggered.




====  <span  style="color: purple;">''Write Elog Message''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Default:'''  "n"
</div>




Write Elog Message


BOOL
If this key in the [[#<class-name> subtree|<class-name> subtree]] is
set to "y", a message will be written to the Elog when an alarm with this  [[#Alarm class|alarm class]] is triggered.


If set to "y" a message will be written to the Elog when alarm is triggered






<br>
---------
<br>


System message interval


INT


Interval in seconds between successive system messages when alarm is triggered
====  <span  style="color: purple;">''System message interval''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:'''  60
</div>
This key in the [[#<class-name> subtree|<class-name> subtree]] contains the
interval in seconds between successive system messages when an alarm with this  [[#Alarm class|alarm class]] is triggered.


Set this key to 0 to ensure that every alarm is recorded into the [[Message System#MIDAS Log file|MIDAS log file]].


'''See important note ''' on the [[Alarm System#Implementation of the MIDAS Alarm System|implementation of this key]] by the alarm system.


<br>
---------
<br>


System message last
====  <span  style="color: purple;">''System message last''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' DWORD
* '''Default:'''  0
</div>


DWORD
This key in the [[#<class-name> subtree|<class-name> subtree]] contains the time when the last alarm system
message was written. It is filled by the alarm system.


Filled by System...


<br>
---------
<br>






Execute command
====  <span  style="color: purple;">''Execute command''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:'''  ""
</div>


STRING
This key in the [[#<class-name> subtree|<class-name> subtree]] may contain a
command to be executed when an alarm with this  [[#Alarm class|alarm class]] is triggered.
This parameter can be used to trigger [[#Alarm triggers Email or SMS|Email or SMS]] alerts.


Command to be executed when alarm is triggered.




<br>
---------
<br>




Execute last
====  <span  style="color: purple;">''Execute interval''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' INT
* '''Default:'''  0
</div>


DWORD
When an alarm with this  [[#Alarm class|alarm class]] is triggered, if a valid
[[#Execute command|Execute command]] is supplied, the command will be repeated at an interval in seconds designated by this key in the [[#<class-name> subtree|<class-name> subtree]], providing the interval set is greater than 0.








<br>
---------
<br>


Stop run
====  <span  style="color: purple;">''Execute last''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' DWORD
* '''Default:'''  0
</div>


BOOL
This key in the [[#<class-name> subtree|<class-name> subtree]]
contains the time when the alarm system last executed the command in the key [[#Execute command|Execute command]].
It is filled by the alarm system.




<br>
---------
<br>






Display BGColor
====  <span  style="color: purple;">''Stop run''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' BOOL
* '''Default:'''  "n"
</div>


STRING
This key in the [[#<class-name> subtree|<class-name> subtree]]  if set to "y"
will cause the run to stop when an alarm with this [[#Alarm class|alarm class]] is triggered.


Background colour of alarm banner (mhttpd only).


<br>
---------
<br>






Display FGColor
====  <span  style="color: purple;">''Display BGColor''</span> ====
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* '''Type:''' STRING
* '''Default:'''  "red"
</div>


STRING
This key in the [[#<class-name> subtree|<class-name> subtree]] contains the
background colour of the alarm banner on the [[mhttpd]]  [[Status Page]]. The alarm banner
appears if the alarm is triggered.


Foreground colour of alarm banner (mhttpd only).




<br>
---------
<br>


Examples of an /Alarms tree


Part of the /Alarms tree is shown below using odbedit (see also mhttpd Alarm page).


[local:pol:S]/>cd /alarms
====  <span  style="color: purple;">''Display FGColor''</span> ====
[local:pol:S]/Alarms>ls
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
Alarm system active            y
* '''Type:''' STRING
Alarms
* '''Default:''' "black"
Classes
</div>
Some of the types of alarm under the /Alarms/Alarms tree for an experiment are shown below:
 
[local:pol:S]/Alarms>ls -r -lt
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Alarms                          DIR
    Alarm system active        BOOL    1    4    4h  0  RWD  y
    Alarms                      DIR
        Demo ODB                DIR
            Active              BOOL    1    4    >99d 0  RWD  n
            Triggered          INT    1    4    >99d 0  RWD  0
            Type                INT    1    4    >99d 0  RWD  3
            Check interval      INT    1    4    >99d 0  RWD  60
            Checked last        DWORD  1    4    >99d 0  RWD  0
            Time triggered firstSTRING  1    32    >99d 0  RWD
            Time triggered last STRING  1    32    >99d 0  RWD
            Condition          STRING  1    256  >99d 0  RWD  /Runinfo/Run number > 100
            Alarm Class        STRING  1    32    >99d 0  RWD  Alarm
            Alarm Message      STRING  1    80    >99d 0  RWD  Run number became too large
        Demo periodic          DIR
              Active              BOOL    1    4    >99d 0  RWD  n
            Triggered          INT    1    4    >99d 0  RWD  0
            Type               INT    1    4    >99d 0  RWD  4
            Check interval      INT    1    4    >99d 0  RWD  28800
            Checked last        DWORD  1    4    >99d 0  RWD  1058817867
            Time triggered firstSTRING  1    32    >99d 0  RWD
            Time triggered last STRING  1    32    >99d 0  RWD
            Condition          STRING  1    256  >99d 0  RWD
            Alarm Class        STRING  1    32    >99d 0  RWD  Warning
            Alarm Message      STRING  1    80    >99d 0  RWD  Please do your shift checks
        fePOL                  DIR
            Active              BOOL    1    4    19s  0  RWD  y
            Triggered          INT    1    4    19s  0  RWD  205
            Type                INT    1    4    3s  0  RWD  2
            Check interval      INT    1    4    19s  0  RWD  60
            Checked last        DWORD  1    4    19s  0  RWD  1259196026
            Time triggered firstSTRING  1    32    19s  0  RWD  Wed Nov 25 12:59:33 2009
            Time triggered last STRING 1    32    19s  0  RWD  Wed Nov 25 16:40:26 2009
            Condition          STRING  1    256  3s  0  RWD  Program not running
            Alarm Class        STRING  1    32    19s  0  RWD  Caution
            Alarm Message      STRING  1    80    19s  0  RWD  Program fePOL is not running
        thr2 trip              DIR
            Active              BOOL    1    4    3s  0  RWD  y
            Triggered          INT    1    4    3s  0  RWD  0
            Type                INT    1    4    3s  0  RWD  3
            Check interval      INT    1    4    3s  0  RWD  15
            Checked last        DWORD  1    4    3s  0  RWD  1259196042
            Time triggered firstSTRING  1    32    3s  0  RWD
            Time triggered last STRING  1    32    3s  0  RWD
            Condition          STRING  1    256  3s  0  RWD  /Equipment/Info ODB/Variables/last failed thr test = 2
            Alarm Class        STRING  1    32    3s  0  RWD  Threshold
            Alarm Message      STRING  1    80    3s  0  RWD  Laser threshold check failed
In the above example,
 
Demo odb and Demo periodic were created when the ODB was created.
The alarm Fepol was added automatically when the user filled the alarm class field in the /Programs/fepol sub-tree.
The other alarm thr2_trip was added by the user.
Four Classes of alarms (Alarm, Caution, Warning and Threshold) are defined under the /Alarms/Classes tree for this experiment. Alarm and Warning were created when the ODB was created. The user added two more classes, Caution and Threshold, by copying and editing one of the existing classes. The Classes defined for the experiment are shown below:
 
  Classes                      DIR
        Alarm                  DIR
            Write system messageBOOL    1    4    27h  0  RWD  y
            Write Elog message  BOOL    1    4    27h  0  RWD  n
            System message interINT    1    4    27h  0  RWD  60
            System message last DWORD  1    4    27h  0  RWD  0
            Execute command    STRING  1    256  27h  0  RWD
            Execute interval    INT    1    4    27h  0  RWD  0
            Execute last        DWORD  1    4    27h  0  RWD  0
            Stop run            BOOL    1    4    27h  0  RWD  n
            Display BGColor    STRING  1    32    27h  0  RWD  red
            Display FGColor    STRING  1    32    27h  0  RWD  black
        Warning                DIR
            Write system messageBOOL    1    4    >99d 0  RWD  y
            Write Elog message  BOOL    1    4    >99d 0  RWD  n
            System message interINT    1    4    >99d 0  RWD  60
            System message last DWORD  1    4    >99d 0  RWD  0
            Execute command    STRING  1    256  >99d 0  RWD
            Execute interval    INT    1    4    >99d 0  RWD  0
            Execute last        DWORD  1    4    >99d 0  RWD  0
            Stop run            BOOL    1    4    >99d 0  RWD  n
            Display BGColor    STRING  1    32    >99d 0  RWD  red
            Display FGColor    STRING  1    32    >99d 0  RWD  black
      Caution                DIR
            Write system messageBOOL    1    4    19s  0  RWD  y
            Write Elog message  BOOL    1    4    19s  0  RWD  n
            System message interINT    1    4    19s  0  RWD  60
            System message last DWORD  1    4    19s  0  RWD  1259196026
            Execute command    STRING  1    256  19s  0  RWD
            Execute interval    INT    1    4    19s  0  RWD  0
            Execute last        DWORD  1    4    19s  0  RWD  0
            Stop run            BOOL    1    4    19s  0  RWD  y
            Display BGColor    STRING  1    32    19s  0  RWD  blue
            Display FGColor    STRING  1    32    19s  0  RWD  red
      Threshold              DIR
            Write system messageBOOL    1    4    >99d 0  RWD  n
            Write Elog message  BOOL    1    4    >99d 0  RWD  n
            System message interINT    1    4    >99d 0  RWD  60
            System message last DWORD  1    4    >99d 0  RWD  0
            Execute command    STRING  1    256  >99d 0  RWD
            Execute interval    INT    1    4    >99d 0  RWD  0
            Execute last        DWORD  1    4    >99d 0  RWD  0
            Stop run            BOOL    1    4    >99d 0  RWD  n
            Display BGColor    STRING  1    32    >99d 0  RWD  yellow
            Display FGColor    STRING  1    32    >99d 0  RWD  black
 
 
Alarm triggers Email or SMS alerts
 
It is also possible to have the MIDAS alarm system send email or SMS alerts to cell phones when alarms are triggered. This can be configured by defining an ODB alarm on a critical ODB parameter, e.g.
 
/Alarms/Alarms/Liquid Level
Active                  y
Triggered                0 (0x0)
Type                    3 (0x3)
Check interval          60 (0x3C)
Checked last    1227690148 (0x492D10A4)
Time triggered first    (empty)
Time triggered last    (empty)
Condition              /Equipment/Environment/Variables/Input[0] < 10
Alarm Class            Level Alarm
Alarm Message          Liquid Level is only %s
In this example, the alarm triggers an alarm of class "Level Alarm". This alarm class is defined as follows:
 
/Alarms/Classes/Level Alarm
Write system message    y
Write Elog message      n
System message interval 600 (0x258)
System message last    0 (0x0)
Execute command        /home/midas/level_alarm '%s'
Execute interval        1800 (0x708)
Execute last            0 (0x0)
Stop run                n
Display BGColor        red
Display FGColor        black
The key here is to call a script "level_alarm", which can send emails. Use something like:
 
#/bin/csh
echo $1 | mail -s \"Level Alarm\" your.name@domain.edu
odbedit -c 'msg 2 level_alarm \"Alarm was sent to your.name@domain.edu\"'
The second command just generates a MIDAS system message for confirmation. Most cell phones (depends on the provider) have an email address. If you send an email there, it will be translated into a SMS message.
 
The script file above can of course be more complicated. A perl script could be used that parses an address list, so other interested parties can register by adding his/her email address to that list. The script may also collects some other slow control variables (like pressure, temperature) and combine them into the SMS message.
 
For very sensitive systems, having an alarm via SMS may not be sufficient, since the alarm system could be down (e.g. computer crash, network failure). In this case 'negative alarms' can be used. For example, every 30 minutes the system may send an SMS with the current parameter values. If the expected message is not received, it may indicate that something in the MIDAS system is wrong.
 
Implementation of the MIDAS Alarm System
 
Alarms are checked inside alarm.c::al_check(). This function is called by cm_yield() every 10 seconds and by rpc_server_thread(), also every 10 seconds. For remote MIDAS clients, their al_check() issues an RPC_AL_CHECK RPC call into the MIDAS server utility mserver, where rpc_server_dispatch() calls the local al_check(). As result, all alarm checks run inside a process directly attached to the local MIDAS shared memory (inside a local client or inside an mserver process for a remote client). Each and every MIDAS client runs the alarm checks. To prevent race conditions between different MIDAS clients, access to al_check() is serialized using the ALARM semaphore. Inside al_check(), alarms are triggered using al_trigger_alarm(), which in turn calls al_trigger_class(). Inside al_trigger_class(), the alarm is recorded into an elog or into midas.log using cm_msg(MTALK).
 
Special note should be made of the ODB setting "/Alarm/Classes/xxx/System message interval", which has a surprising effect - after an alarm is recorded into system messages (using cm_msg(MTALK)), no record is made of any subsequent alarms until the time interval set by this variable elapses. With default value of 60 seconds, after one alarm, no more alarms are recorded for 60 seconds. Also, because all the alarms are checked at the same time, only the first triggered alarm will be recorded.
 
As of alarm.c rev 4683, "System message interval" is set to 0 ensures that every alarm is recorded into the MIDAS log file. (In previous revisions, this setting may still miss some alarms).
 
There are 3 types of alarms:
 
1) "program not running" alarms.
 
These alarms are enabled in ODB by setting /Programs/ppp/Alarm class. Each time al_check() runs, every program listed in /Programs is tested using "cm_exist()" and if the program is not running, the time of first failure is remembered in /Programs/ppp/First failed.
 
If the program has not been running for longer than the time set in ODB key /Programs/ppp/Check interval, an alarm is triggered (if enabled by /Programs/ppp/Alarm class and the program is restarted (if enabled by /Programs/ppp/Auto restart).
 
The "not running" condition is tested every 10 seconds (each time al_check() is called), but the frequency of "program not running" alarms can be reduced by increasing the value of /Alarms/Alarms/ppp/Check interval (default value 60 seconds). This can be useful if System message interval is set to zero.
 
2) "evaluated" alarms


3) "periodic" alarms
This key in the [[#<class-name> subtree|<class-name> subtree]] contains the
foreground colour of the alarm banner on the [[mhttpd]] [[Status Page]].  The alarm banner
appears if the alarm is triggered.


There is nothing surprising in these alarms. Each alarm is checked with a time period set by /Alarm/xxx/Check interval. The value of an evaluated alarm is computed using al_evaluate_condition().
[[Category:ODB Tree]]
[[Category:Alarms]]

Latest revision as of 15:57, 6 August 2015


Links

Purpose

The ODB /Alarms tree contains user and system information related to alarms (see Alarm System). The information from this tree is displayed in the Alarms Page.

Creating the /Alarms tree

When the ODB is created, the /Alarms tree is automatically created, with

These are shown in the example below.

The user may create other alarms and classes by copying an existing <alarm-name> or <alarm-class> subtree and editing (via Alarms Page or odbedit) as required.

By default, the Alarm System is NOT active. When it is active, the overall alarm is checked once every minute. Once the alarm has been triggered, the message associated with the alarm can be repeated at a different rate.


Example

This example (using odbedit) shows the default /Alarms tree. See also Alarms Page.

$ odbedit
[local:pol:S]/>cd /alarms
[local:pol:S]/Alarms>ls
Alarm system active             y
Alarms
Classes
[local:pol:S]/Alarms>ls -r -lt
Key name                        Type    #Val  Size  Last Opn Mode Value
---------------------------------------------------------------------------
Alarms                          DIR
   Alarm system active         BOOL    1     4     4h   0   RWD  y
   Alarms                      DIR
       Demo ODB                DIR
           Active              BOOL    1     4     >99d 0   RWD  n
           Triggered           INT     1     4     >99d 0   RWD  0
           Type                INT     1     4     >99d 0   RWD  3
           Check interval      INT     1     4     >99d 0   RWD  60
           Checked last        DWORD   1     4     >99d 0   RWD  0
           Time triggered firstSTRING  1     32    >99d 0   RWD
           Time triggered last STRING  1     32    >99d 0   RWD
           Condition           STRING  1     256   >99d 0   RWD  /Runinfo/Run number > 100
           Alarm Class         STRING  1     32    >99d 0   RWD  Alarm
           Alarm Message       STRING  1     80    >99d 0   RWD  Run number became too large
       Demo periodic           DIR
             Active              BOOL    1     4     >99d 0   RWD  n
           Triggered           INT     1     4     >99d 0   RWD  0
           Type                INT     1     4     >99d 0   RWD  4
           Check interval      INT     1     4     >99d 0   RWD  28800
           Checked last        DWORD   1     4     >99d 0   RWD  1058817867
           Time triggered firstSTRING  1     32    >99d 0   RWD
           Time triggered last STRING  1     32    >99d 0   RWD
           Condition           STRING  1     256   >99d 0   RWD
           Alarm Class         STRING  1     32    >99d 0   RWD  Warning
           Alarm Message       STRING  1     80    >99d 0   RWD  Please do your shift checks
     fePOL                   DIR
           Active              BOOL    1     4     19s  0   RWD  y
           Triggered           INT     1     4     19s  0   RWD  205
           Type                INT     1     4     3s   0   RWD  2
           Check interval      INT     1     4     19s  0   RWD  60
           Checked last        DWORD   1     4     19s  0   RWD  1259196026
           Time triggered firstSTRING  1     32    19s  0   RWD  Wed Nov 25 12:59:33 2009
           Time triggered last STRING  1     32    19s  0   RWD  Wed Nov 25 16:40:26 2009
           Condition           STRING  1     256   3s   0   RWD  Program not running
           Alarm Class         STRING  1     32    19s  0   RWD  Caution
           Alarm Message       STRING  1     80    19s  0   RWD  Program fePOL is not running
 Classes                      DIR
       Alarm                   DIR
           Write system messageBOOL    1     4     27h  0   RWD  y
           Write Elog message  BOOL    1     4     27h  0   RWD  n
           System message interINT     1     4     27h  0   RWD  60
           System message last DWORD   1     4     27h  0   RWD  0
           Execute command     STRING  1     256   27h  0   RWD
           Execute interval    INT     1     4     27h  0   RWD  0
           Execute last        DWORD   1     4     27h  0   RWD  0
           Stop run            BOOL    1     4     27h  0   RWD  n
           Display BGColor     STRING  1     32    27h  0   RWD  red
           Display FGColor     STRING  1     32    27h  0   RWD  black
       Warning                 DIR
           Write system messageBOOL    1     4     >99d 0   RWD  y
           Write Elog message  BOOL    1     4     >99d 0   RWD  n
           System message interINT     1     4     >99d 0   RWD  60
           System message last DWORD   1     4     >99d 0   RWD  0
           Execute command     STRING  1     256   >99d 0   RWD
           Execute interval    INT     1     4     >99d 0   RWD  0
           Execute last        DWORD   1     4     >99d 0   RWD  0
           Stop run            BOOL    1     4     >99d 0   RWD  n
           Display BGColor     STRING  1     32    >99d 0   RWD  red
           Display FGColor     STRING  1     32    >99d 0   RWD  black


/Alarms tree structure

The /Alarms ODB tree is split into 2 subtrees:

  • Alarms subtree containing as many alarms (i.e. <alarm-name> subtrees) as the user desires. Each alarm must be one of the four defined Alarm Types. These also define the conditions that trigger the alarm.
  • Classes subtree containing as many classes (i.e. <class-name> subtrees) as the user desires. Each class defines the action to be taken when the alarm occurs. Two Classes (Alarm and Warning) are defined by default (see Example).

See also Alarm System.



Keys in the /Alarms ODB tree

Alarm system active

  • Type: BOOL
  • Default: "n"

If this key in the /Alarms ODB tree is set to "y", the alarm system is active. Set to "n" to deactivate.





Alarms subtree

  • Type: DIR


Sub-tree in the /Alarms ODB tree defining each individual alarm condition.






<alarm-name> subtree

  • Type: DIR


This subtree in the /Alarms/alarms subtree specifies the name of one of the defined alarms, where name is substituted for <alarm-name>. This subtree will be repeated with a different name for each defined alarm. In the example two <alarm-name> subtrees are shown, called "demo odb" and "demo periodic"






Active

  • Type: BOOL
  • Default: "n"

If this key in the <alarm-name> subtree is set to "y" , this particular alarm is active.







Triggered

  • Type: INT
  • Default: 0

If this key in the <alarm-name> subtree is non-zero, this alarm has been triggered. Filled by System.







Type

  • Type: INT
  • Default:

This key in the <alarm-name> subtree must be one of the listed Alarm Types.






Check interval

  • Type: INT
  • Default: 60


This key in the <alarm-name> subtree contains the frequency in seconds at which this alarm condition is to be checked by the alarm system.






Checked last

  • Type: DWORD
  • Default:


This key in the <alarm-name> subtree is written by the Alarm System.







Time triggered first

  • Type: STRING
  • Default:


This key in the <alarm-name> subtree is written by the Alarm System.






Time triggered last

  • Type: STRING
  • Default:


This key in the <alarm-name> subtree is written by the Alarm System.







Condition

  • Type: STRING
  • Default:


This key in the <alarm-name> subtree contains the condition on which alarm should trigger for evaluated alarms. See #Evaluated Alarm conditions.






Alarm class

  • Type: STRING
  • Default:


This key in the <alarm-name> subtree is set to one of the existing Alarm classes, e.g. the defaults "Alarm","Warning" or a user-defined class, e.g. "Caution". The Alarm class must be defined in the Classes subtree.





Alarm message

  • Type: STRING
  • Default:

This key in the <alarm-name> subtree constains the message to be written when alarm triggers.






Classes subtree

  • Type: DIR

Sub-tree in the /Alarms ODB tree defining each individual alarm class. Each class defines the individual action to be performed by a predefined and requested alarm.





<class-name> subtree

  • Type: DIR

This subtree in the Classes subtree defines one of the alarm classes of name name, where name is substituted for <class-name>. This subtree will be repeated with a different name for each defined class. In the example above, two <class-name> subtrees are shown, named "Alarm" and "Warning".






Write System Message

  • Type: BOOL
  • Default: "y"


If this key in the <class-name> subtree is set to "y", a message will be sent to the System log when an alarm with this alarm class is triggered.





Write Elog Message

  • Type: BOOL
  • Default: "n"


If this key in the <class-name> subtree is set to "y", a message will be written to the Elog when an alarm with this alarm class is triggered.







System message interval

  • Type: INT
  • Default: 60

This key in the <class-name> subtree contains the interval in seconds between successive system messages when an alarm with this alarm class is triggered.

Set this key to 0 to ensure that every alarm is recorded into the MIDAS log file.

See important note on the implementation of this key by the alarm system.




System message last

  • Type: DWORD
  • Default: 0

This key in the <class-name> subtree contains the time when the last alarm system message was written. It is filled by the alarm system.






Execute command

  • Type: STRING
  • Default: ""

This key in the <class-name> subtree may contain a command to be executed when an alarm with this alarm class is triggered. This parameter can be used to trigger Email or SMS alerts.






Execute interval

  • Type: INT
  • Default: 0

When an alarm with this alarm class is triggered, if a valid Execute command is supplied, the command will be repeated at an interval in seconds designated by this key in the <class-name> subtree, providing the interval set is greater than 0.






Execute last

  • Type: DWORD
  • Default: 0

This key in the <class-name> subtree contains the time when the alarm system last executed the command in the key Execute command. It is filled by the alarm system.






Stop run

  • Type: BOOL
  • Default: "n"

This key in the <class-name> subtree if set to "y" will cause the run to stop when an alarm with this alarm class is triggered.






Display BGColor

  • Type: STRING
  • Default: "red"

This key in the <class-name> subtree contains the background colour of the alarm banner on the mhttpd Status Page. The alarm banner appears if the alarm is triggered.






Display FGColor

  • Type: STRING
  • Default: "black"

This key in the <class-name> subtree contains the foreground colour of the alarm banner on the mhttpd Status Page. The alarm banner appears if the alarm is triggered.