Overview
Connect Prometheus Alertmanager to Callgoose SQIBS. When an alert fires, SQIBS opens an incident and notifies the on-call engineer. When the alert resolves, SQIBS resolves the incident. You can also let SQIBS run an automation workflow to fix known issues on its own.
How it works
firing opens it, resolved closes it.Want SQIBS to fix known issues before anyone is paged? See Fix known issues automatically.
Step 1: Add the API filter in SQIBS
- In SQIBS, open Services, choose the service that should receive these incidents, and click Add Or Update API Integration.
- In Select Integration Template, choose Prometheus. It fills in a filter that works for Alertmanager as it is:
| Part | Payload JSON key | Key value contains | What it does |
|---|---|---|---|
| Trigger | "status" |
firing |
Opens an incident. |
| Resolve | "status" |
resolved |
Resolves the open incident. |
| Match | "groupKey" |
Ties the resolve message to the incident it opened. |
- Choose the Priority for these incidents and click Save.
The template is a starting point, not a lock-in: every value stays visible and you can change it. It opens an incident for every firing alert. To be more selective, for example only alerts with severity critical, add a condition with the JSON key creator.
In the API Integration window, click the round
{} button at the bottom right. Paste a payload that SQIBS received (copy it from the API log). SQIBS lists every key with its value, each with a copy button. For example, it shows "alerts".[0]."labels"."severity" with the value critical. Copy the key into Payload JSON key and the value into Key value contains. No need to read the JSON by hand.More filter ideas: API filter examples for Prometheus.
Step 2: Add the SQIBS receiver to Alertmanager
Edit alertmanager.yml. Add a receiver named callgoose-sqibs and make it the default receiver of your route, or use it in any route you choose:
route:
group_by: ['alertname']
group_wait: 10s
group_interval: 30s
repeat_interval: 10m
receiver: 'callgoose-sqibs'
receivers:
- name: 'callgoose-sqibs'
webhook_configs:
- url: 'https://<your-endpoint>/sqibs-api/v1/process?from=prometheus&token=<your-token>'
send_resolved: true
Replace <your-endpoint> and <your-token> with the values from your SQIBS account. See how to get the endpoint URL.
send_resolved: true. Without it Alertmanager never tells SQIBS that the alert is over, and the incident stays open.Alertmanager already sends JSON, so nothing needs converting. Reload Alertmanager after saving the file.
Step 3: Add alert rules in Prometheus
Any alert rule works. These two are a simple start, using metrics from node_exporter. Save them in a rules file, for example node_alerts.yml, and list it under rule_files in prometheus.yml:
groups:
- name: node_alerts
rules:
- alert: HostDown
expr: up == 0
for: 1m
labels:
severity: critical
annotations:
summary: "Host is down - {{ $labels.instance }}"
description: "{{ $labels.instance }} ({{ $labels.job }}) has been down for more than 1 minute."
- alert: DiskSpaceLow
expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes * 100 <= 10
for: 5m
labels:
severity: warning
annotations:
summary: "Disk space low - {{ $labels.instance }}"
description: "{{ $labels.mountpoint }} on {{ $labels.instance }} has {{ $value | humanize }}% free."
Step 4: Test it
The API log saves payloads only while API Tracking is on for the token. Turn it on under API Tokens before you test; it switches itself off after 48 hours.
- Stop
node_exporteron one test host. - After about a minute, the Alerts page in Prometheus shows
HostDownas firing, and Alertmanager shows the alert. - In SQIBS, a new incident opens. Its description holds the payload Alertmanager sent, for example:
{
"receiver": "callgoose-sqibs",
"status": "firing",
"alerts": [{
"status": "firing",
"labels": { "alertname": "HostDown", "instance": "10.0.0.15:9100", "job": "node", "severity": "critical" },
"annotations": { "summary": "Host is down - 10.0.0.15:9100", "description": "10.0.0.15:9100 (node) has been down for more than 1 minute." },
"startsAt": "2026-11-21T10:13:01.537Z"
}],
"groupKey": "{}:{alertname=\"HostDown\"}",
"commonLabels": { "alertname": "HostDown", "severity": "critical" },
"version": "4"
}
- Start
node_exporteragain. Alertmanager sends the same alert with statusresolved, and SQIBS resolves the incident.
No incident? Open the API log (with API Tracking on). If the payload is there, the filter did not match it: paste it into the JSON key creator and check the keys. If it is not there, check the URL and token in alertmanager.yml.
Fix known issues automatically
Many alerts have a known fix: restart a service, clear a full disk, run a health check. SQIBS can run that fix for you the moment the incident opens. This is event-driven auto remediation.
Safe by design. SQIBS runs only workflows your own engineers wrote and approved for your environment. Nothing is guessed or generated: the same steps an engineer would run by hand, run sooner. If the workflow does not fix the issue, the engineer is paged as usual.
To set it up, in the same API filter:
- Select Workflow: choose the automation workflow to run when this filter opens an incident.
- If success then make incident resolved: turn on to resolve the incident when the workflow succeeds.
- Waiting time for escalate to escalation policies: how many minutes to give the workflow before paging the on-call engineer.
Start with low-risk workflows, such as a health check, and add fixes as you come to trust them.
Learn more:
- Prometheus: incident auto remediation, a full worked example.
- Automation workflow: build a workflow from actions.
- Run workflows where your systems are: SSH, Bash, PowerShell, Python, Ansible, kubectl, Terraform.