Overview

Connect Prometheus Alertmanager to Callgoose SQIBS. When an alert fires, SQIBS opens an incident and notifies the on-call engineer. When the alert resolves, SQIBS resolves the incident. You can also let SQIBS run an automation workflow to fix known issues on its own.

You need
Prometheus with Alertmanager, and a SQIBS API token
Setup time
About 10 minutes
Result
Incidents open and resolve on their own

How it works

Prometheus
Alert rule fires
For example, a host stops answering for one minute.
Alertmanager
Sends a webhook
To your SQIBS API endpoint, as JSON.
SQIBS
API filter matches
The filter decides whether to open or resolve, and the priority.
SQIBS
Incident opens
Status firing opens it, resolved closes it.

Want SQIBS to fix known issues before anyone is paged? See Fix known issues automatically.

Step 1: Add the API filter in SQIBS

  1. In SQIBS, open Services, choose the service that should receive these incidents, and click Add Or Update API Integration.
  2. In Select Integration Template, choose Prometheus. It fills in a filter that works for Alertmanager as it is:
Part Payload JSON key Key value contains What it does
Trigger "status" firing Opens an incident.
Resolve "status" resolved Resolves the open incident.
Match "groupKey" Ties the resolve message to the incident it opened.
  1. Choose the Priority for these incidents and click Save.

The template is a starting point, not a lock-in: every value stays visible and you can change it. It opens an incident for every firing alert. To be more selective, for example only alerts with severity critical, add a condition with the JSON key creator.

Find the right key with the JSON key creator
In the API Integration window, click the round {} button at the bottom right. Paste a payload that SQIBS received (copy it from the API log). SQIBS lists every key with its value, each with a copy button. For example, it shows "alerts".[0]."labels"."severity" with the value critical. Copy the key into Payload JSON key and the value into Key value contains. No need to read the JSON by hand.

More filter ideas: API filter examples for Prometheus.

Step 2: Add the SQIBS receiver to Alertmanager

Edit alertmanager.yml. Add a receiver named callgoose-sqibs and make it the default receiver of your route, or use it in any route you choose:

route:
  group_by: ['alertname']
  group_wait: 10s
  group_interval: 30s
  repeat_interval: 10m
  receiver: 'callgoose-sqibs'

receivers:
  - name: 'callgoose-sqibs'
    webhook_configs:
      - url: 'https://<your-endpoint>/sqibs-api/v1/process?from=prometheus&token=<your-token>'
        send_resolved: true

Replace <your-endpoint> and <your-token> with the values from your SQIBS account. See how to get the endpoint URL.

Keep send_resolved: true. Without it Alertmanager never tells SQIBS that the alert is over, and the incident stays open.

Alertmanager already sends JSON, so nothing needs converting. Reload Alertmanager after saving the file.

Step 3: Add alert rules in Prometheus

Any alert rule works. These two are a simple start, using metrics from node_exporter. Save them in a rules file, for example node_alerts.yml, and list it under rule_files in prometheus.yml:

groups:
  - name: node_alerts
    rules:
      - alert: HostDown
        expr: up == 0
        for: 1m
        labels:
          severity: critical
        annotations:
          summary: "Host is down - {{ $labels.instance }}"
          description: "{{ $labels.instance }} ({{ $labels.job }}) has been down for more than 1 minute."
      - alert: DiskSpaceLow
        expr: node_filesystem_avail_bytes{fstype!~"tmpfs|overlay"} / node_filesystem_size_bytes * 100 <= 10
        for: 5m
        labels:
          severity: warning
        annotations:
          summary: "Disk space low - {{ $labels.instance }}"
          description: "{{ $labels.mountpoint }} on {{ $labels.instance }} has {{ $value | humanize }}% free."

Step 4: Test it

The API log saves payloads only while API Tracking is on for the token. Turn it on under API Tokens before you test; it switches itself off after 48 hours.

  1. Stop node_exporter on one test host.
  2. After about a minute, the Alerts page in Prometheus shows HostDown as firing, and Alertmanager shows the alert.
  3. In SQIBS, a new incident opens. Its description holds the payload Alertmanager sent, for example:
{
  "receiver": "callgoose-sqibs",
  "status": "firing",
  "alerts": [{
    "status": "firing",
    "labels": { "alertname": "HostDown", "instance": "10.0.0.15:9100", "job": "node", "severity": "critical" },
    "annotations": { "summary": "Host is down - 10.0.0.15:9100", "description": "10.0.0.15:9100 (node) has been down for more than 1 minute." },
    "startsAt": "2026-11-21T10:13:01.537Z"
  }],
  "groupKey": "{}:{alertname=\"HostDown\"}",
  "commonLabels": { "alertname": "HostDown", "severity": "critical" },
  "version": "4"
}
  1. Start node_exporter again. Alertmanager sends the same alert with status resolved, and SQIBS resolves the incident.

No incident? Open the API log (with API Tracking on). If the payload is there, the filter did not match it: paste it into the JSON key creator and check the keys. If it is not there, check the URL and token in alertmanager.yml.

Fix known issues automatically

Many alerts have a known fix: restart a service, clear a full disk, run a health check. SQIBS can run that fix for you the moment the incident opens. This is event-driven auto remediation.

Incident opens
Workflow starts
SQIBS runs the automation workflow linked to the filter.
Fixed
Incident resolved
The issue is fixed before anyone has to step in.
Not fixed
Escalated to the engineer
After the waiting time you set, the escalation policy pages on-call as usual.

Safe by design. SQIBS runs only workflows your own engineers wrote and approved for your environment. Nothing is guessed or generated: the same steps an engineer would run by hand, run sooner. If the workflow does not fix the issue, the engineer is paged as usual.

To set it up, in the same API filter:

  1. Select Workflow: choose the automation workflow to run when this filter opens an incident.
  2. If success then make incident resolved: turn on to resolve the incident when the workflow succeeds.
  3. Waiting time for escalate to escalation policies: how many minutes to give the workflow before paging the on-call engineer.

Start with low-risk workflows, such as a health check, and add fixes as you come to trust them.

Learn more: