Alert hooks for repeated check failures and recovery.
The alert tracker keeps a consecutive-failure counter per check id.
When a check fails threshold times in a row (default 3), the
subscriber is notified with notify(:failed, result); when the check
recovers (returns ok after having failed), the subscriber is notified
with notify(:recovered, result).
Subscribers
A subscriber is any module implementing the Botica.Alerts.Subscriber
behaviour (or a plain fun.(kind, result)).
Example
defmodule MyApp.Slack do
@behaviour Botica.Alerts.Subscriber
@impl true
def notify(:failed, result) do
# POST to Slack
:ok
end
def notify(:recovered, result), do: :ok
end
Botica.Alerts.subscribe(MyApp.Slack)
Botica.Alerts.track(check_id, result) # call after every runBehaviour
defmodule Botica.Alerts.Subscriber do
@callback notify(kind :: :failed | :recovered, result :: map()) :: :ok | {:error, term()}
endSummary
Functions
Returns a specification to start this module under a supervisor.
Returns the current consecutive failure count for a check.
Formats an alert line with a severity-specific prefix.
Subscriber behaviour.
Resets the failure counter for a check (e.g. after manual intervention).
Registers a subscriber (module implementing the behaviour or a
fun.(kind, result)).
Updates the consecutive-failure counter for a check and fires alerts.
Removes a subscriber.
Functions
Returns a specification to start this module under a supervisor.
See Supervisor.
@spec failures(atom()) :: non_neg_integer()
Returns the current consecutive failure count for a check.
Formats an alert line with a severity-specific prefix.
Severity levels
:info— blue[i]prefix:warning— yellow[!]prefix:error— red[x]prefix:critical— red[X]prefix (with double border)
Examples
iex> Botica.Alerts.format_alert(:warning, "Postgres latency > 1s")
"[!] Postgres latency > 1s"
iex> Botica.Alerts.format_alert(:critical, "Disk full")
"[X] Disk full"Subscriber behaviour.
@spec reset(atom()) :: :ok
Resets the failure counter for a check (e.g. after manual intervention).
Registers a subscriber (module implementing the behaviour or a
fun.(kind, result)).
Updates the consecutive-failure counter for a check and fires alerts.
Call this after every check run with the check result:
Botica.Alerts.track(result.id, result)A check that fails threshold consecutive times triggers
notify(:failed, result). The first ok after a failure streak
triggers notify(:recovered, result).
Returns :ok always (alerts are fire-and-forget).
Removes a subscriber.