Overview
This guide builds event-driven auto remediation end to end. Prometheus checks a website; when it stops answering, Alertmanager tells Callgoose SQIBS. SQIBS opens an incident and immediately runs an automation workflow that restarts the web server. If the site is back, SQIBS resolves the incident; if not, it escalates to the on-call engineer as usual.
How it works
Safe by design. The workflow runs only the script you wrote and approved, with arguments you set. Nothing is guessed. If it cannot fix the problem, the engineer is paged as usual.
Step 1: Run Prometheus, Alertmanager and the Blackbox exporter
Any Prometheus setup works. This example runs all three in containers with Podman (Docker works the same way). Create these four files in one folder.
blackbox.yml, the HTTP check:
modules:
http_2xx:
prober: http
timeout: 5s
http:
preferred_ip_protocol: ip4
valid_status_codes: [200, 301, 302]
prometheus.yml, which probes your website (replace https://www.example.com):
global:
scrape_interval: 15s
evaluation_interval: 15s
alerting:
alertmanagers:
- static_configs:
- targets: ['alertmanager:9093']
rule_files:
- /etc/prometheus/alert_rules.yml
scrape_configs:
- job_name: blackbox
metrics_path: /probe
params:
module: [http_2xx]
static_configs:
- targets: ['https://www.example.com']
relabel_configs:
- source_labels: [__address__]
target_label: __param_target
- source_labels: [__param_target]
target_label: instance
- target_label: __address__
replacement: blackbox_exporter:9115
alert_rules.yml, which fires WebsiteDown after one minute without an answer:
groups:
- name: website
rules:
- alert: WebsiteDown
expr: probe_success == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Website {{ $labels.instance }} is down"
description: "{{ $labels.instance }} has not answered for more than 1 minute.
alertmanager.yml, which sends alerts to SQIBS, resolved ones included:
route:
receiver: callgoose-sqibs
group_by: ['alertname', 'instance']
group_wait: 10s
group_interval: 30s
repeat_interval: 10m
receivers:
- name: callgoose-sqibs
webhook_configs:
- url: 'https://<your-endpoint>/sqibs-api/v1/process?from=prometheus&token=<your-token>'
send_resolved: true
Replace <your-endpoint> and <your-token> with the values from your SQIBS account. See how to get the endpoint URL and API tokens.
Start them:
podman network create monitoring
podman run -d --name blackbox_exporter --network monitoring -p 9115:9115 \
-v ./blackbox.yml:/etc/blackbox_exporter/config.yml:Z prom/blackbox-exporter
podman run -d --name alertmanager --network monitoring -p 9093:9093 \
-v ./alertmanager.yml:/etc/alertmanager/alertmanager.yml:Z prom/alertmanager
podman run -d --name prometheus --network monitoring -p 9090:9090 \
-v ./prometheus.yml:/etc/prometheus/prometheus.yml:Z \
-v ./alert_rules.yml:/etc/prometheus/alert_rules.yml:Z prom/prometheus
Open http://<prometheus-host>:9090/targets and check that the website target is UP.
Step 2: Prepare the automation runner
The workflow runs on a Callgoose automation runner, a small program inside your network that connects out to SQIBS, collects jobs and runs them. It needs no inbound firewall rule. See the automation runner and the firewall requirements.
- In SQIBS, open Profiles (under automation), choose your team and click Add Profile. Set the profile type to Runner and give it a name.
- In the Download key dialog, download the credentials key for the runner. Keep it safe: it lets the runner sign in to your account.
- In the runner profile, use Download Runner JAR, and install the runner on a Linux host with the key (full steps in the runner guide).
- In the runner profile's Internal Profiles, add a Bash profile, so the runner can run Bash actions.
- The runner host must reach the web server over SSH with a key, as the user the runner runs as. See passwordless SSH. Allow that user to run
sudo systemctl restart nginxwithout a password, and nothing more.
Prefer not to run a runner? SQIBS can also connect to your server directly over SSH. The profiles page shows the key to add as an authorized SSH key on your server, and the firewall requirements page lists the addresses to allow. The runner is the better choice for servers that are not reachable from the internet.
Step 3: Create the action
- In SQIBS, open Actions, choose your team and click Add Action. Choose Bash and the runner profile.
- Name it
restart_weband describe what it does. Write the script in the Editor or use File Upload:
#!/bin/bash
# Usage: restart_web.sh <ssh_user> <server> <ssh_port> <service> <url>
# Checks the website; if it does not answer, restarts the web service over SSH and checks again.
set -u
user=$1 server=$2 port=$3 service=$4 url=$5
check() { curl -fsS -o /dev/null --max-time 10 "$url"; }
if check; then echo "website answers, nothing to do"; exit 0; fi
echo "website down, restarting $service on $server"
ssh -p "$port" -o BatchMode=yes "$user@$server" "sudo systemctl restart $service" || exit 2
sleep 5
if check; then echo "fixed: website answers again"; exit 0; fi
echo "still down after restart"; exit 1
The exit code tells SQIBS the result: 0 means fixed, anything else means not fixed. You can also start from one of the ready-made scripts under CALLGOOSE ACTIONS (use Copy from) and adapt it.
3. Save the action.
Step 4: Create the incident workflow
- Open Incident Workflow, choose your team and click Add Workflow.
- Name it
Restart website, set the Category to REMEDY, and add therestart_webaction. - Add the Arguments in order: SSH user, server, SSH port, service and website URL, for example
autouser,web-01.example.com,22,nginx,https://www.example.com. - Save the workflow.
Step 5: Link the workflow to the Prometheus API filter
- Open Services, choose the service and click Add Or Update API Integration.
- In Select Integration Template, choose Prometheus (see the Prometheus integration for its values).
- To run this workflow only for this alert, add a second content check to the trigger: key
"alerts".[0]."labels"."alertname"containsWebsiteDown. - Select Workflow:
Restart website. - Turn on If success then make incident resolved.
- Set Waiting time for escalate to escalation policies, for example
4minutes: enough for the restart and the check. - Click Save.
In the API Integration window, click the round
{} button at the bottom right. Paste a payload that SQIBS received (copy it from the API log). SQIBS lists every key with its value, each with a copy button. Copy the key into Payload JSON key and the value into Key value contains. No need to read the JSON by hand.Step 6: Test it
- On the web server, stop nginx:
sudo systemctl stop nginx. - After about a minute, Prometheus fires
WebsiteDownand SQIBS opens an incident. - The workflow runs on the runner, restarts nginx and checks the site.
- Open the incident and select the Workflow Log tab to see each action and its output.
- The script exits
0, so SQIBS resolves the incident. Prometheus then sends the resolved alert as well.
To see the escalation path, make the fix fail (for example, point the action at a service name that does not exist). The incident stays open, and when the waiting time ends the escalation policy pages on-call.
No incident? Open the API log (with API Tracking on). If the payload is there, the filter did not match it: paste it into the JSON key creator and check the keys. If it is not there, check the URL and token in the Alertmanager receiver.
Next steps
- Add workflows for your other common alerts: disk full, service stopped, certificate renewal.
- Use a DIAGNOSTICS workflow to collect logs and add them to the incident before anyone looks at it.
- Run other kinds of actions: SSH, PowerShell, Python, Ansible, kubectl, Terraform.