Overview
OpenTelemetry collects logs, metrics and traces and sends them to a backend. It does not decide when something is wrong: alert rules live in the backend that stores the data. So to get incidents from OpenTelemetry data, set up alerts in that backend and send them to SQIBS.
How it works
Want SQIBS to fix known issues before anyone is paged? See Fix known issues automatically.
Choose your backend
Follow the SQIBS page for the backend that stores your OpenTelemetry data:
- Metrics in Prometheus: Prometheus (Alertmanager).
- Dashboards and alerts in Grafana (Mimir, Loki, Tempo): Grafana.
- OpenTelemetry-native backends: SigNoz, Uptrace, HyperDX, Honeycomb, OpenObserve.
- Commercial platforms: Datadog, New Relic, Dynatrace, Splunk Observability.
Do not point an OpenTelemetry exporter at the SQIBS API. Exporters send every span, metric and log line in OTLP format; SQIBS expects one JSON message per alert.
Fix known issues automatically
Many alerts have a known fix: restart a service, clear a full disk, run a health check. SQIBS can run that fix for you the moment the incident opens. This is event-driven auto remediation. For example, when an alert on your trace data shows checkout latency rising, SQIBS can run a workflow that scales out the checkout service.
Safe by design. SQIBS runs only workflows your own engineers wrote and approved for your environment. Nothing is guessed or generated: the same steps an engineer would run by hand, run sooner. If the workflow does not fix the issue, the engineer is paged as usual.
To set it up, in the same API filter:
- Select Workflow: choose the automation workflow to run when this filter opens an incident.
- If success then make incident resolved: turn on to resolve the incident when the workflow succeeds.
- Waiting time for escalate to escalation policies: how many minutes to give the workflow before paging the on-call engineer.
Start with low-risk workflows, such as a health check, and add fixes as you come to trust them.
Learn more:
- Prometheus: incident auto remediation, a full worked example.
- Automation workflow: build a workflow from actions.
- Run workflows where your systems are: SSH, Bash, PowerShell, Python, Ansible, kubectl, Terraform.