Overview

This guide builds event-driven auto remediation end to end. Prometheus checks a website; when it stops answering, Alertmanager tells Callgoose SQIBS. SQIBS opens an incident and immediately runs an automation workflow that restarts the web server. If the site is back, SQIBS resolves the incident; if not, it escalates to the on-call engineer as usual.

You need
A Linux host for Prometheus, a server running nginx, and SQIBS with automation
Setup time
About 45 minutes
Result
A down website is restarted and the incident closed, without paging anyone

How it works

Step 1
Prometheus alerts
Alertmanager posts the firing alert to SQIBS.
Step 2
Filter matches
The API filter opens an incident and starts the linked workflow.
Step 3
Workflow runs
A runner in your network runs your action: check the site, restart the web service.
Step 4
Resolve or escalate
Success resolves the incident; otherwise on-call is paged after the waiting time.

Safe by design. The workflow runs only the script you wrote and approved, with arguments you set. Nothing is guessed. If it cannot fix the problem, the engineer is paged as usual.

Step 1: Run Prometheus, Alertmanager and the Blackbox exporter

Any Prometheus setup works. This example runs all three in containers with Podman (Docker works the same way). Create these four files in one folder.

blackbox.yml, the HTTP check:

modules:
  http_2xx:
    prober: http
    timeout: 5s
    http:
      preferred_ip_protocol: ip4
      valid_status_codes: [200, 301, 302]

prometheus.yml, which probes your website (replace https://www.example.com):

global:
  scrape_interval: 15s
  evaluation_interval: 15s
alerting:
  alertmanagers:
    - static_configs:
        - targets: ['alertmanager:9093']
rule_files:
  - /etc/prometheus/alert_rules.yml
scrape_configs:
  - job_name: blackbox
    metrics_path: /probe
    params:
      module: [http_2xx]
    static_configs:
      - targets: ['https://www.example.com']
    relabel_configs:
      - source_labels: [__address__]
        target_label: __param_target
      - source_labels: [__param_target]
        target_label: instance
      - target_label: __address__
        replacement: blackbox_exporter:9115

alert_rules.yml, which fires WebsiteDown after one minute without an answer:

groups:
  - name: website
    rules:
      - alert: WebsiteDown
        expr: probe_success == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Website {{ $labels.instance }} is down"
          description: "{{ $labels.instance }} has not answered for more than 1 minute.

alertmanager.yml, which sends alerts to SQIBS, resolved ones included:

route:
  receiver: callgoose-sqibs
  group_by: ['alertname', 'instance']
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 10m
receivers:
  - name: callgoose-sqibs
    webhook_configs:
      - url: 'https://<your-endpoint>/sqibs-api/v1/process?from=prometheus&token=<your-token>'
        send_resolved: true

Replace <your-endpoint> and <your-token> with the values from your SQIBS account. See how to get the endpoint URL and API tokens.

Start them:

podman network create monitoring

podman run -d --name blackbox_exporter --network monitoring -p 9115:9115 \
  -v ./blackbox.yml:/etc/blackbox_exporter/config.yml:Z prom/blackbox-exporter

podman run -d --name alertmanager --network monitoring -p 9093:9093 \
  -v ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:Z prom/alertmanager

podman run -d --name prometheus --network monitoring -p 9090:9090 \
  -v ./prometheus.yml:/etc/prometheus/prometheus.yml:Z \
  -v ./alert_rules.yml:/etc/prometheus/alert_rules.yml:Z prom/prometheus

Open http://<prometheus-host>:9090/targets and check that the website target is UP.

Step 2: Prepare the automation runner

The workflow runs on a Callgoose automation runner, a small program inside your network that connects out to SQIBS, collects jobs and runs them. It needs no inbound firewall rule. See the automation runner and the firewall requirements.

  1. In SQIBS, open Profiles (under automation), choose your team and click Add Profile. Set the profile type to Runner and give it a name.
  2. In the Download key dialog, download the credentials key for the runner. Keep it safe: it lets the runner sign in to your account.
  3. In the runner profile, use Download Runner JAR, and install the runner on a Linux host with the key (full steps in the runner guide).
  4. In the runner profile's Internal Profiles, add a Bash profile, so the runner can run Bash actions.
  5. The runner host must reach the web server over SSH with a key, as the user the runner runs as. See passwordless SSH. Allow that user to run sudo systemctl restart nginx without a password, and nothing more.

Prefer not to run a runner? SQIBS can also connect to your server directly over SSH. The profiles page shows the key to add as an authorized SSH key on your server, and the firewall requirements page lists the addresses to allow. The runner is the better choice for servers that are not reachable from the internet.

Step 3: Create the action

  1. In SQIBS, open Actions, choose your team and click Add Action. Choose Bash and the runner profile.
  2. Name it restart_web and describe what it does. Write the script in the Editor or use File Upload:
#!/bin/bash
# Usage: restart_web.sh <ssh_user> <server> <ssh_port> <service> <url>
# Checks the website; if it does not answer, restarts the web service over SSH and checks again.
set -u
user=$1 server=$2 port=$3 service=$4 url=$5

check() { curl -fsS -o /dev/null --max-time 10 "$url"; }

if check; then echo "website answers, nothing to do"; exit 0; fi
echo "website down, restarting $service on $server"
ssh -p "$port" -o BatchMode=yes "$user@$server" "sudo systemctl restart $service" || exit 2
sleep 5
if check; then echo "fixed: website answers again"; exit 0; fi
echo "still down after restart"; exit 1

The exit code tells SQIBS the result: 0 means fixed, anything else means not fixed. You can also start from one of the ready-made scripts under CALLGOOSE ACTIONS (use Copy from) and adapt it. 3. Save the action.

Step 4: Create the incident workflow

  1. Open Incident Workflow, choose your team and click Add Workflow.
  2. Name it Restart website, set the Category to REMEDY, and add the restart_web action.
  3. Add the Arguments in order: SSH user, server, SSH port, service and website URL, for example autouser, web-01.example.com, 22, nginx, https://www.example.com.
  4. Save the workflow.
  1. Open Services, choose the service and click Add Or Update API Integration.
  2. In Select Integration Template, choose Prometheus (see the Prometheus integration for its values).
  3. To run this workflow only for this alert, add a second content check to the trigger: key "alerts".[0]."labels"."alertname" contains WebsiteDown.
  4. Select Workflow: Restart website.
  5. Turn on If success then make incident resolved.
  6. Set Waiting time for escalate to escalation policies, for example 4 minutes: enough for the restart and the check.
  7. Click Save.
Find the right key with the JSON key creator
In the API Integration window, click the round {} button at the bottom right. Paste a payload that SQIBS received (copy it from the API log). SQIBS lists every key with its value, each with a copy button. Copy the key into Payload JSON key and the value into Key value contains. No need to read the JSON by hand.

Step 6: Test it

  1. On the web server, stop nginx: sudo systemctl stop nginx.
  2. After about a minute, Prometheus fires WebsiteDown and SQIBS opens an incident.
  3. The workflow runs on the runner, restarts nginx and checks the site.
  4. Open the incident and select the Workflow Log tab to see each action and its output.
  5. The script exits 0, so SQIBS resolves the incident. Prometheus then sends the resolved alert as well.

To see the escalation path, make the fix fail (for example, point the action at a service name that does not exist). The incident stays open, and when the waiting time ends the escalation policy pages on-call.

No incident? Open the API log (with API Tracking on). If the payload is there, the filter did not match it: paste it into the JSON key creator and check the keys. If it is not there, check the URL and token in the Alertmanager receiver.

Next steps

  • Add workflows for your other common alerts: disk full, service stopped, certificate renewal.
  • Use a DIAGNOSTICS workflow to collect logs and add them to the incident before anyone looks at it.
  • Run other kinds of actions: SSH, PowerShell, Python, Ansible, kubectl, Terraform.