Alarm System: Difference between revisions

From MidasWiki
Jump to navigation Jump to search
(Added messagebird voice calling instructions)
(The midas documentation has moved to https://midas.triumf.ca/docs/)
Tag: Replaced
 
Line 1: Line 1:
{{Pagelinks}}
The midas documentation has moved. The new documentation about the alarm system is in:
 
* [https://midas.triumf.ca/docs/develop/features/monitoring_and_control/alarms/ Alarms: overview] (alarms and alarm classes, web interfaces)
 
* [https://midas.triumf.ca/docs/develop/features/monitoring_and_control/alarms/configuring/ Alarms: configuring] (alarm types, and scripts that send email, Slack, Telegram, Mattermost or phone alerts)
= Links =
* [https://midas.triumf.ca/docs/develop/features/monitoring_and_control/alarms/programming/ Alarms: programming] (implementation details and the alarm functions)
<div style="column-count:3;-moz-column-count:3;-webkit-column-count:3">
* [[/Alarms ODB tree]]
* [[Alarms Page]]
* [[Mhttpd|mhttpd MIDAS web server]]
</div>
 
= Introduction =
MIDAS provides an alarm system, which by default is turned off. When the alarm system is [[/Alarms ODB tree#Alarm system active|activated]] and an alarm condition is detected, alarm messages are sent by the system which appear as an alarm banner on the [[Status Page|mhttpd status page]], and as a message on any windows running [[odbedit]] clients. The alarm system is flexible and can be extensively customized for each experiment
using the [[Alarms Page|mhttpd Alarms Page]] or [[odbedit]].
 
The alarm system is built-in and part of the main experiment scheduler. This means no separate task is necessary to benefit from the alarm system. Its setup and activation is done through the '''[[/Alarms ODB tree]]'''. The alarm system includes several other features such as sequencing and control of the experiment. The alarm capabilities are:
* Alarm setting on any ODB variable against a threshold parameter.
* Alarm triggered by evaluated condition
* Selection of Alarm check frequency
* Selection of Alarm trigger frequency
* Customization alarm scheme; under this scheme multiple choices of alarm type can be selected
* Selection of alarm message destination (to system message log or to elog)
* email or SMS alerts can be sent
* Alarm triggered when a Program is not running
 
 
= Implementation of the MIDAS Alarm System =
 
The alarm system source code is [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/alarm_8c_source.html alarm.c].
Alarms are checked inside [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__alfunctioncode.html alarm.c::al_check()]. This function is called by cm_yield() every 10 seconds and by rpc_server_thread(), also every 10 seconds. For remote MIDAS clients, their al_check() issues an RPC_AL_CHECK RPC call into the MIDAS server utility [[mserver]], where rpc_server_dispatch() calls the local al_check(). As result, all alarm checks run inside a process directly attached to the local MIDAS shared memory (inside a local client or inside an mserver process for a remote client). Each and every MIDAS client runs the alarm checks. To prevent race conditions between different MIDAS clients, access to al_check() is serialized using the ALARM semaphore. Inside al_check(), alarms are triggered using
[http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__alfunctioncode.html al_trigger_alarm()], which in turn calls al_trigger_class(). Inside al_trigger_class(), the alarm is recorded into an elog or into [[Message System|midas.log]] using cm_msg(MTALK).
 
Special note should be made of the ODB setting [[/Alarms ODB tree#System message interval|system message interval]], which has a surprising effect - after an alarm is recorded into system messages (using cm_msg(MTALK)), no record is made of any subsequent alarms until the time interval set by this variable elapses. With default value of 60 seconds, after one alarm, no more alarms are recorded for 60 seconds. Also, because all the alarms are checked at the same time, '''only the first''' triggered alarm will be recorded.
 
As of alarm.c rev 4683, {{Odbpath|path=/Alarms/System message interval}} is set to 0 ensures that every alarm is recorded into the [[Message System#MIDAS Log file|MIDAS log file]]. (In previous revisions, this setting may still miss some alarms).
 
 
 
 
<div id="Alarm class"></div>
= Alarms structure =
The [[/Alarms ODB tree]] structure is split into 2 sections:
*"Alarms" which define the condition to be tested. The user can create as many [[/Alarms ODB tree#Alarms subtree|Alarms]] as desired, but each must be one of the four defined [[#Alarm Types|Alarm types]] .
*"Classes" which define the action to be taken when the alarm occurs. Two Classes (Alarm and Warning) are defined by default. The user can add more [[/Alarms ODB tree#Classes subtree|Classes]] as desired.
<br><br>
 
In order to make the system flexible, each alarm class may perform different actions when an alarm is given. For example, it may
* write a system message (see [[/Alarms ODB tree#Write System Message|Write System Message]])
* write to the elog (see [[/Alarms ODB tree#Write Elog Message|Write Elog Message]])
* stop the run (see [[/Alarms ODB tree#Stop run|Stop run]])
* spawn a detached script listed in the ODB variable [[/Alarms ODB tree#Execute command|Execute command]]. This feature is used when an Alarm triggers Email or SMS alerts (see [[#Alarm triggering Email or SMS alerts|example]]).
<br>
 
= Alarm Types =
 
The four available Alarm Types are shown in Table 1. They are defined in [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/html/midas_8h.html&nbsp midas.h]. The alarm type is entered into the [[/Alarms ODB tree#Type|Type]] key.
 
 
{|  style="text-align: left; width: 100%; background-color: white;" border="3" cellpadding="2" cellspacing="2"
|+ Table 1 : Defined Alarm Types
|-
| colspan="2" rowspan="1" style="vertical-align: top; background-color: lavender; font-weight: bold;" | Alarm Type
| colspan="1" rowspan="1" style="vertical-align: top; background-color: lavender; font-weight: bold;" | INT value
| colspan="1" rowspan="1" style="vertical-align: top; background-color: lavender; font-weight: bold;" | Explanation
 
|- 
| colspan="1" rowspan="1"  style="vertical-align: top; background-color: white; font-weight: bold;"  |Internal alarms
| colspan="1" rowspan="1"  style="vertical-align: top; background-color: white; font-weight: normal;" | AT_INTERNAL
|1
|Trigger on internal (program) alarm setting through the use of the al_...() functions.
 
|- 
| style="vertical-align: top; background-color: white; font-weight: bold;" |Program alarms
|AT_PROGRAM
|2
|Triggered on condition of the state of the defined task (i.e. program not running)
 
|- 
| style="vertical-align: top; background-color: white; font-weight: bold;" |Evaluated alarms
|AT_EVALUATED
|3
|Triggered by ODB value on given arithmetical condition.
 
|- 
| style="vertical-align: top; background-color: white; font-weight: bold;" |Periodic alarms
|AT_PERIODIC
|4
|Triggered by timeout condition defined in the alarm setting.
 
|}
 
 
==  Program Alarm ==
 
Program (or rather "Program not running") alarms, when enabled, warn the user when a program is not running.
 
Program alarms are enabled by setting the ODB key [[/Programs ODB tree#Alarm class|/Programs/<client-name>/Alarm class]] to a valid Alarm class specified in the [[/Alarms ODB tree]]. The first time the alarm is triggered, an <span style="color: purple; font-style:italic;">/Alarms/Alarms/<client-name></span> subtree will be created automatically. The program alarm will not be visible in the [[Alarms Page]] until the alarm has triggered, and the subtree created.
 
The alarm system periodically calls [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__alfunctioncode.html al_check()]. This causes every client listed in the {{Odbpath|path=/Programs}} ODB tree to be tested using [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/doc/html/group__cmfunctionc.html cm_exist()] to see if it is running. If the client is not running, the time of first failure is recorded in the ODB key [[/Programs ODB tree#First failed|/Programs/<client-name>/First failed]].
 
If the client has not been running for longer than the time set in ODB key [[/Programs ODB tree#Check interval|/Programs/<client-name>/Check interval]], a "Program not running" alarm is triggered (if enabled by [[/Programs ODB tree#Alarm class|Alarm class]]) and the program is restarted (if enabled by [[/Programs ODB tree#Auto restart|/Programs/<client-name>/Auto restart]] and a valid [[/Programs ODB tree#Start command|Start command]] is supplied).
 
The "not running" condition is tested every 10 seconds (each time al_check() is called), but the frequency of ''Program not running'' alarms can be reduced by increasing the value of the ODB key
[[/Programs ODB tree#Check interval|/Programs/<client-name>/Check interval]]
(default value 60 seconds). This can be useful if  [[/Alarms ODB tree#System message interval|System message interval]] in the specified alarm class subtree is set to zero.
 
 
== Periodic Alarm ==
The periodic alarm is activated periodically according to the time in [[/Programs ODB tree#Check interval|/Programs/<client-name>/Check interval]]. An example of a periodic alarm is "Demo Periodic" in the  [[/Alarms ODB tree#Example|example]].
 
== Evaluated Alarm  ==
 
Evaluated alarms require an ''alarm condition'' which is entered into the ODB key [[/Alarms ODB tree#Condition|Condition]] in the  [[/Alarms ODB tree#<alarm_name> subtree|<alarm_name> subtree]].
The condition may be simply a '''comparison''' between any ODB variable and a threshold parameter, e.g.
 
/Runinfo/Run number > 100
or it may be an '''evaluated condition'''. One can write conditions like
 
  /Equipment/HV/Variables/Input[*] > 100
or
 
  /Equipment/HV/Variables/Input[2-3] > 100
or
 
  /Equipment/HV/Variables/Input[1,4,5-8,10] > 100
 
to check all or certain values from an array. A dash means a range including both indices. If one array element fulfills the alarm condition, the alarm is triggered. In addition, bit-wise alarm conditions are possible, e.g.
 
  /Equipment/Environment/Variables/Input[0] & 8
The alarm is triggered if bit #3 is set in Input[0].
 
The value of an evaluated alarm is computed using al_evaluate_condition() in [http://ladd00.triumf.ca/~daqweb/doc/midas-devel/html/alarm_8c_source.html alarm.c].
 
Sometimes alarm trigger kind of accidentally on some analog input voltage if they are noise. In order to avoid this, the "Trigger count required" can be used. If this value contains a non-zero value of N, then the alarm system requires the alarm condition to be met N consecutive times until the alarm to be triggered.
 
== Internal Alarm ==
These are triggered in a program using a call to
[https://daq.triumf.ca/~daqweb/doc/midas-devel/html/group__alfunctioncode.html al_trigger_alarm()]. See also description of al_trigger_alarm() sequence  [[#Implementation of the MIDAS Alarm System|above]].
 
 
There is nothing surprising in these alarms. Each alarm is checked with a time period set by ODB key [[ /Alarms ODB tree#Check interval|Check interval]] in the [[/Alarms ODB tree]].
 
= Alarm triggering Email or SMS alerts =
 
It is possible to have the MIDAS alarm system send email or SMS alerts to cell phones when alarms are triggered. This can be configured by defining an ODB alarm on a critical ODB parameter, e.g.
 
/Alarms/Alarms/Liquid Level
Active                  y
Triggered                0 (0x0)
Type                    3 (0x3)
Check interval          60 (0x3C)
Checked last    1227690148 (0x492D10A4)
Trigger count            0 (0x0)
Trigger count required  0 (0x0)
Time triggered first    (empty)
Time triggered last    (empty)
Condition              /Equipment/Environment/Variables/Input[0] < 10
Alarm Class            Level Alarm
Alarm Message          Liquid Level is only %s
 
In this example, the alarm triggers an alarm of class "Level Alarm". This alarm class is defined as follows:
 
/Alarms/Classes/Level Alarm
Write system message    y
Write Elog message      n
System message interval 600 (0x258)
System message last    0 (0x0)
Execute command        /home/midas/level_alarm '%s'
Execute interval        1800 (0x708)
Execute last            0 (0x0)
Stop run                n
Display BGColor        red
Display FGColor        black
 
The key here is to call a script "level_alarm", which can send emails. Use something like:
 
#/bin/csh
echo $1 | mail -s \"Level Alarm\" your.name@domain.edu
odbedit -c 'msg 2 level_alarm \"Alarm was sent to your.name@domain.edu\"'
 
Ensure the shell script is executable (e.g., <code>chmod +x script.sh</code>).
 
The second command just generates a MIDAS system message for confirmation. Most cell phones (depends on the provider) have an email address. If you send an email there, it will be translated into a SMS message.
 
The script file above can of course be more complicated. A perl script could be used that parses an address list, so other interested parties can register by adding his/her email address to that list. The script may also collects some other slow control variables (like pressure, temperature) and combine them into the SMS message.
 
For very sensitive systems, having an alarm via SMS may not be sufficient, since the alarm system could be down (e.g. computer crash, network failure). In this case 'negative alarms' can be used. For example, every 30 minutes the system may send an SMS with the current parameter values. If the expected message is not received, it may indicate that something in the MIDAS system is wrong.
 
= Alarm triggering Slack notifications =
 
A more modern way for notification is to use messengers apps such as Slack. To send alarms to Slack, do the following:
 
* Go to https://api.slack.com/apps
* Create an App "MIDAS alarms", select "From scratch"
* Name it "MIDAS alarm" or similar, select your workspace
* Click on "Add features and functionality"
* Select "Incoming Webhooks"
* Activate Incoming Webhooks
* Click on "Add New Webhook to Workspace"
* Select channel where alarms get posted, allow access
* Copy sample curl request and replace "Hello World" by "$1", such as:
 
curl -X POST -H 'Content-type: application/json' --data "{\"text\":\"$1\"}' https://hooks.slack.com/services/[xxx]/[yyy]/[zzz]
 
(leave they keys xxx, yyy, zzz as shown on the web page).
 
* Call the curl command from the alarm system by putting the above command under <code>/Alarms/Classes/All/Execute command</code> into the ODB.
* Ensure the shell script is executable (e.g., <code>chmod +x script.sh</code>)
 
= Alarm triggering Telegram notifications =
 
A telegram bot needs to be created to post Midas alarms into telegram channels.
 
* Go to https://web.telegram.org and login by scanning the QR-code with the telegram app on your phone
* Go to https://t.me/botfather
* Click on "open in web"
* A chat window will open.
* Send "/newbot" into the chat and assign name and username to your bot
* At the end of the bot creation a access token will be displayed. Save that access token.
* Post "/mybots" into the chat
* Select the bot that was just created
* Click on "Bot settings"
* Click on "Allow Groups"
* Enable groups for the bot
* Click on "back to settings"
* Click on Channel Admin Rights
* Click on "Manage channel" (a checkmark should appear next to the text)
* Close this chat.
 
* Add the created bot to a telegram group chat via searching for the bot name in "add members" (same as adding new People to the group)
* Now you need the chat_id of the same group chat. When you open the group chat in a webbrowser the URL should look like this https://web.telegram.org/a/#-1700000000 (where 1700000000 will be a different number for you)
* The group chat_id is then -1001700000000 (add -100 in front of the number in your URL)
* With the chat_id from above and the access token from the bot creation process you can post to the telegram group like this: (replace <chat_id> and <accessToken> with your numbers)
 
  curl -s -X  POST "https://api.telegram.org/bot<accessToken>/sendMessage" -d chat_id=<chat_id> -d text="🚨 Midas alarm triggered with message = $1"
 
* Create a shell script with this command
* Ensure the shell script is executable (e.g., <code>chmod +x script.sh</code>)
* Call the shell script from the alarm system by putting <code>path/to/script.sh '%1$s'</code> under <code>/Alarms/Classes/All/Execute command</code> into the ODB.
* Should you want to post the same alarm to multiple systems you can just separate scripts in <code>/Alarms/Classes/All/Execute command</code> with a semicolon <code>path/to/script.sh '%1$s';path/to/script2.sh '%1$s'</code>
 
= Alarm triggering Mattermost notifications =
 
First a mattermost bot needs to be created. To do this you need to be an Admin of the Team that the bot should post to.
 
* In the drop down menu on the top left corner of the mattermost page click on "Integrations" (note: this field will be missing if you are not an Admin of the selected mattermost team)
* Select "Incoming Webhooks"
* Click on "Add Incoming Webhook"
* Give the bot a title and username (e.g. midasalarms) and select the channel to post to, click "save"
* Now a URL for the incoming webhook should be displayed (https://mattermost.gitlab.rlp.net/hooks/<token>). Copy that URL into the curl command in the following script:
 
 
  #!/bin/bash
  alarm_message=$1
  generate_post_data()
  {
    cat <<EOF
    {
    "text": "@all Midas alarm triggered with message = $alarm_message",
    "icon_emoji":":rotating_light:",
    "username":"Midas"}
  EOF
  }
  curl -i -X POST -H 'Content-Type: application/jsoni' --data "$(generate_post_data)" https://mattermost.gitlab.rlp.net/hooks/<token>
 
- Call the shell script from the alarm system by putting <code>path/to/script.sh '%1$s'</code> under <code>/Alarms/Classes/All/Execute command</code> into the ODB.
Ensure the shell script is executable (e.g., <code>chmod +x script.sh</code>).
 
 
= Alarm triggering MessageBird Voice Call =
 
We will use a text-to-voice to read out MIDAS alarms in a phone call to shifters. The service is called bird (formerly messagebird). Note that bird hosts two platforms: "engagement" and "connectivity". Each has its own API, documentation, and web interface. We will use the engagement platform. Be sure to look at the [https://docs.bird.com correct documentation].
 
== MessageBird Setup ==
 
# Make an account: https://app.bird.com/
# Add some funds to your main wallet: https://app.bird.com/settings/billing/overview/wallets/
# Buy a number. This number will be the where the call originates from, so you should pick something local to your area. 
## In the left-hand navigation side panel: "Developer" -> "Numbers" -> "Numbers & Senders"
## Top-right: "Browse Inventory"
## Make sure your number is capable of voice calling (it seems like most of them are)
## Follow the on-screen prompts to finish your purchase and activate the number. You will have to create a UseCase. Either "Mixed" or "Security Alert" should do.
# Ensure compliance with local regulations
## Go back to the number summary: "Developer" -> "Numbers" -> "Numbers & Senders". You should now have you number listed. Click on it.
## At the top there is a menu, select "Voice"
## You should see some links to set up brand identification and compliance (what will you be using the number for, the company associated with the number, etc). Follow the on-screen instructions. Once compliance has been requested it may take a few days to be approved.
## While you wait, you also need to create a Voice Channel. In this same menu, click on "+ Create channel". You can see your channels from the side bar navigation: "Developer" -> "Voice" -> "Voice Setup".
 
== Information Needed to Make Calls ==
 
You need a few pieces of info:
 
* API Access Key
** In the side bar navigation: "Developer" -> "API Access Keys" (at the bottom)
** Make a new key. Keep it somewhere safe.
* Workspace ID
** In the side bar navigation: "Settings" (at the bottom) -> "Workspaces" -> Click on your workspace
* Channel ID
** In the side bar navigation: "Developer" -> "Voice" -> "Voice Setup" -> Click on your voice channel -> Settings (top right gear)
* Call ID
** Only available once the call is placed. See the script below.
* Outgoing number
** In the side bar navigation: "Developer" -> "Numbers" -> "Numbers & Senders"
 
== Placing a Call ==
 
You can follow the documentation here: https://docs.bird.com/api/voice-api/voice-calls-api. Here is a python example script:
<pre>
import json, requests, time
 
# settings
workspaceId = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
channelId = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
APIKEY = 'YourAPIKey'
outgoing = '16042221047' # outgoing number with country code
targetnumber = 'YourPurcahsedPhoneNumber' # who to call
message = "Hello, world! This is my first message."
 
# place a call
# https://docs.bird.com/api/voice-api/voice-calls-api/initiate-an-outbound-call
def place_call(number):
 
    # place the call
    response = requests.post(
        f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls",
        headers={"Authorization":f"AccessKey {APIKEY}","Content-Type":"application/json"},
        data=json.dumps({
        "from": outgoing,
        "to": number,
        "ringTimeout": 60,
        "maxDuration": 120,
        "record": False,
        })
    )
 
    # call id is needed for the next steps
    data = response.json()
    callId = data['id']
    return callId
 
# get the call status
# https://docs.bird.com/api/voice-api/voice-calls-api/get-a-call-insights
def get_call_status(callId):
    response = requests.get(
        f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls/{callId}/insights",
        headers={"Authorization":f"AccessKey {APIKEY}","Accept":"*/*"},
    )
    data = response.json()
    return data['status']
 
# text-to-voice: play message in the call
# https://docs.bird.com/api/voice-api/voice-calls-api/say-text-to-speech-tts
def play_message(callId, message):
    response = requests.post(
        f"https://api.bird.com/workspaces/{workspaceId}/channels/{channelId}/calls/{callId}/say",
        headers={"Authorization":f"AccessKey {APIKEY}","Content-Type":"application/json"},
        data=json.dumps({
        "text": message,
        "locale": "en-US",
        "voice": "text",
        "loop": 10, # repeat the message this many times
        "timeout": 30
        })
    )
    return response.json()
 
# RUN =========================
 
# place call
callId = place_call(targetnumber)
 
# wait until call is answered, or until 30s has passed
t0 = time.time()
while time.time() - t0 < 30:
    status = get_call_status(callId)
    print(f'Call status: "{status}"')
 
    # call was answered
    if status == 'ongoing':
        break
 
    # check again later
    time.sleep(1)
 
# play the message
time.sleep(0.5)
play_message(callId, message+',') # comma after message introduces a pause between loops
</pre>
 
[[Category:Alarms]]

Latest revision as of 22:00, 30 October 2026

The midas documentation has moved. The new documentation about the alarm system is in:

Retrieved from "https://daq00.triumf.ca/MidasWiki/index.php?title=Alarm_System&oldid=3669"