Creating Alerts

Hemdal allows you to monitor hosts and services by running periodic checks and transitioning through defined states based on the command result.

Alert Structure

An alert is configured with the following properties:

• id: A unique identifier string (e.g. UUID) for the alert process.
• name: A human-readable name describing what is being monitored.
• enabled: Boolean flag (true by default). When false, the alert is in :disabled state and performs no checks.
• check_in_sec: Interval in seconds between checks while in the :normal (healthy) state (default: 60).
• recheck_in_sec: Interval in seconds between retries when a check fails (default: 5).
• broken_recheck_in_sec: Interval in seconds between checks once confirmed :broken (default: 30).
• retries: Number of consecutive failures in :failing state before transitioning to :broken (default: 3).
• command_args: A list of string arguments passed to the executed command (default: []).
• host: The host configuration on which the command will be executed (see Hemdal.Config.Host).
• command: The command to execute (see Hemdal.Config.Command).
• notifiers: A list of notifiers to dispatch notifications to upon state transitions (see Hemdal.Config.Notifier).

State Machine Lifecycle

Each alert runs as an independent state machine with four possible states:

1. disabled: The alert is inactive. It does not perform checks until re-enabled.
2. normal: The check succeeds (OK). Checked every check_in_sec seconds.
3. failing: The check returned an error or warning (WARN / FAIL / UNKNOWN). Retried every recheck_in_sec seconds for up to retries times.
4. broken: The alert failed all retries and is confirmed broken. Checked every broken_recheck_in_sec seconds until recovery.

Configuration Examples

1. Elixir Environment Backend (Hemdal.Config.Backend.Env)

In your config/config.exs or runtime configuration:

import Config

config :hemdal, :config_module, Hemdal.Config.Backend.Env

config :hemdal, Hemdal.Config, [
  [
    id: "36c16e85-7221-4021-8d6d-89f38a6d136c",
    name: "Disk Space Check",
    enabled: true,
    check_in_sec: 60,
    recheck_in_sec: 5,
    broken_recheck_in_sec: 30,
    retries: 3,
    host: [
      id: "ec8fff22-41c2-4245-8a7b-5157d40c33a7",
      name: "localhost",
      module: Hemdal.Host.Local,
      max_workers: 2
    ],
    command: [
      name: "check_disk",
      type: "line",
      command: "df -h / | awk 'NR==2 {print $5}' | sed 's/%//' | awk '{if ($1 > 90) print \"[\\\"FAIL\\\", \\\"Disk full: \"$1\"%\\\"]\"; else print \"[\\\"OK\\\", \\\"Disk usage: \"$1\"%\\\"]\"}'",
      decode: true
    ],
    notifiers: [
      [
        module: Hemdal.Notifier.Slack,
        token: "https://hooks.slack.com/services/YOUR_SLACK_WEBHOOK",
        username: "Hemdal Bot",
        log_level: "warn"
      ]
    ]
  ]
]

2. JSON Backend (Hemdal.Config.Backend.Json)

Configure file paths in config/config.exs:

import Config

config :hemdal, :config_module, Hemdal.Config.Backend.Json

config :hemdal, Hemdal.Config,
  hosts_file: "config/hosts.json",
  alerts_file: "config/alerts.json"

In config/hosts.json:

[
  {
    "id": "2a8572d4-ceb3-4200-8b29-dd1f21b50e54",
    "name": "127.0.0.1",
    "module": "Hemdal.Host.Local",
    "max_workers": 2
  }
]

In config/alerts.json:

[
  {
    "id": "52d13d6d-f217-4152-965d-cf5f488ceac4",
    "name": "HTTP Health Check",
    "enabled": true,
    "host_id": "2a8572d4-ceb3-4200-8b29-dd1f21b50e54",
    "check_in_sec": 30,
    "recheck_in_sec": 5,
    "broken_recheck_in_sec": 15,
    "retries": 2,
    "command": {
      "name": "curl_health",
      "type": "line",
      "command": "curl -s -f http://localhost:4000/health && echo '[\"OK\", \"healthy\"]' || echo '[\"FAIL\", \"unhealthy\"]'"
    }
  }
]

Command Types

• line: A single command line executed via the host handler.
• script: A multi-line script written to a temporary file on the host, executed, and cleaned up automatically.
• shell: An interactive shell session allowing real-time bidirectional data piping.

Command Output Formats

When decode: true (default), Hemdal parses the output as JSON. The command can output:

1. A JSON array: ["OK", "All systems operational"] or ["FAIL", "High CPU usage"]
2. A JSON map: {"status": "OK", "message": "Healthy", "custom_metric": 42}
3. A JSON string status: "OK" or "FAIL"

Statuses recognized by Hemdal are "OK", "WARN", "FAIL", and "UNKNOWN". Exit code 0 maps to :ok, exit code 1 maps to WARN, and exit code 2 maps to FAIL.

Managing Alerts Programmatically

# Retrieve all running alerts and their current statuses
Hemdal.get_all_alerts()

# Reload configuration from backend (starts new alerts, updates existing, and stops deleted ones)
Hemdal.reload_all()

# Manually start a specific alert by ID from backend configuration
Hemdal.start_alert!("36c16e85-7221-4021-8d6d-89f38a6d136c")

# Stop an alert
Hemdal.Check.stop("36c16e85-7221-4021-8d6d-89f38a6d136c")