Alerting

Uptrace monitors metrics and logs and sends notifications when a monitor fires. Create monitors in the UI or define them in YAML alongside your infrastructure config.

Monitors

The Alerting → Monitors screen lists every monitor kind, including SLO monitors. Each row shows its name, type, and current status. Use New monitor to choose errors, metrics, or SLOs. The metrics form lets you choose a threshold, anomaly, outlier, or forecast detector.

Clicking a monitor opens its detail view: the evaluation history, current alert state, and a chart of the monitored value over time. From here you can edit the monitor definition, mute it for a set duration, or close open alerts manually.

Monitor types

Threshold monitors evaluate a metric query and fire when its value leaves a fixed range. Use them for infrastructure health, HTTP error rates, span latency, or custom metrics.

Anomaly monitors compare a metric series with a model fitted to its history. Use them when the expected range changes over time.

Outlier monitors compare each series with its peers. Group the metric query by an attribute, such as host_name, so the detector has peers to compare.

Forecast monitors project a metric forward and fire when it is expected to cross a limit within the chosen horizon.

Error monitors watch log records matching a set of conditions — severity level, message content, or exception type — and fire the first time a new matching pattern appears. Uptrace automatically creates a default error monitor for ERROR and FATAL severity logs.

SLO monitors measure a service level indicator against a target and evaluate rules over its error budget. Create one through Alerting → Monitors → New monitor → SLOs. The Alerting → SLOs page shows the subjects discovered from telemetry and their remaining budgets. An SLO rule has one of two kinds:

A burn-rate rule says how fast the budget is being spent; a budget-consumed rule says how much of it is gone. Combine them, for example a budget_consumed rule at 0.75 with a low priority to warn and one at 1 with a higher priority to page. Both kinds accept min_total, the smallest event count at which the rule judges. The older sli rule kind is retired: existing SLI rules were converted to burn-rate rules over the same window.

Anomaly, outlier, and forecast monitors

These monitors share the metric monitor shape — metrics, query, column, resolution, num_eval_points, and min_breach_points — and select a detector with the top-level type. A knob that belongs to another type is accepted and ignored.

Anomaly monitors

An anomaly monitor compares a metric series with a model fitted to its own history and fires when the value leaves the expected range. Use it when the normal range changes over time, for example with daily traffic cycles. The training span is derived from the resolution and the seasonality; training_period is accepted and ignored.

monitors:
  - name: Request rate anomaly
    type: anomaly
    metrics:
      - uptrace_tracing_spans as $spans
    query:
      - perMin(histogram_count($spans)) as rpm
      - where _type = "httpserver"
      - group by service_name
    column: rpm
    num_eval_points: 5
    min_breach_points: 3
    sensitivity: medium
    direction: both
    seasonality: auto

Outlier monitors

An outlier monitor compares each series with its peers at the same moment and fires when one series stands out. It fits no model, so it works from the first check. Group the query by an attribute, such as host_name, so the detector has peers to compare.

keep_firing_for stops an alert from closing and reopening while a series hovers near the edge of its peers. If a series departs again after the alert has been quiet for at least half of that span, Uptrace treats it as a new excursion and notifies at once, whatever the repeat interval. PagerDuty receives a resolve and a new trigger, and Opsgenie a close and a new alert, so each opens a new incident that pages again.

monitors:
  - name: CPU outlier host
    type: outlier
    metrics:
      - system_cpu_load_average_15m as $load_avg_15m
      - system_cpu_time as $cpu_time
    query:
      - $load_avg_15m / uniq($cpu_time.cpu) as cpu_util
      - group by host_name
    column:
      name: cpu_util
      unit: utilization
    num_eval_points: 3
    min_breach_points: 2
    direction: above
    min_peers: 4
    keep_firing_for: 45m

Forecast monitors

A forecast monitor projects a metric forward and fires when the projection is expected to cross min_value or max_value within the horizon. Set at least one of the two values. Use it for capacity, such as disk space that runs out in a few days.

monitors:
  - name: Filesystem fills up
    type: forecast
    metrics:
      - system_filesystem_usage as $fs_usage
    query:
      - $fs_usage{state='used'} / $fs_usage as fs_util
      - group by host_name, mountpoint
    column:
      name: fs_util
      unit: utilization
    num_eval_points: 3
    max_value: 0.95
    horizon: 72h

Incidents

An incident groups alerts from one response episode. Open Alerting → Incidents to create an incident from existing alerts, assign it, record notes, and track its status. Automatic grouping and analysis require a configured generative model client.

Incident policy has its own settings page: click the gear icon beside the Incidents title. There you can turn automatic grouping and analysis on or off, set the grouping window, the reopen window, and the quiet period before a resolved incident is closed, and add instructions that steer the grouping and analysis prompts. A zero or empty value inherits the deployment default. The same settings are available through the API at /internal/v1/incidents/{project_id}/settings (GET and PUT; a PUT replaces every field).

Notification channels

Alerts are delivered through configurable notification channels: email, Slack, Google Chat, Mattermost, Telegram, Microsoft Teams, Pushover, PagerDuty, Opsgenie, ServiceNow, incident.io, AlertManager, and custom webhooks.

Each channel supports condition expressions to filter which alerts it receives — a critical Slack channel can ignore low-severity monitors while a catch-all inbox gets everything.

Next steps

• Metric monitors — threshold alerts on infrastructure and tracing metrics
• Error monitors — alerts on log patterns and exceptions
• Notifications — channels, routing conditions, and delivery frequency