Notifications

Notification channels define where alerts are delivered and under what conditions. You can attach multiple channels to a single monitor and use condition expressions to route alerts selectively — for example, sending all alerts to a general channel but routing URGENT monitors to PagerDuty.

Channels

Requires the mailer option to be enabled in the Uptrace config file. Users must have a valid email address set in their profile. Self-hosted only.

Create an Incoming Webhook in your Slack workspace and paste the webhook URL into the channel form. Select the target channel or DM recipient in the Slack app configuration.

Create an Incoming Webhook integration in your Mattermost team and paste the webhook URL into the channel form.

Create a bot via @BotFather and copy the bot token. Add the bot to the target group or channel, then copy the chat ID from the group info.

Create an Incoming Webhook connector in your Teams channel and paste the webhook URL. Alerts are delivered as adaptive cards.

Create a service in PagerDuty with an "Events API v2" integration and copy the integration key. Uptrace creates and resolves incidents automatically.

Create an API integration in Opsgenie and copy the API key. Uptrace sends alerts with the monitor name as the title.

Enter your Prometheus AlertManager endpoint URL. Alerts are forwarded in the Alertmanager v2 format.

Create an Incoming Webhook in your Google Chat space and paste the webhook URL into the channel form.

Create an application in Pushover to get an API token. Provide the token and your user key in the channel form.

Generate an API key in your incident.io account settings and paste it into the channel form. Uptrace creates and resolves incidents automatically.

Uptrace sends a POST request with a JSON body to the configured URL on every alert state change. Use this to integrate with any custom system.

Configure your ServiceNow instance URL and credentials. Uptrace creates and resolves incidents through the ServiceNow REST API.

Create a channel via Alerting → Channels → New channel. Select the channel type, fill in the connection details, and optionally add a routing condition to filter which alerts it receives.

Channel conditions

When creating a channel, you can specify a condition expression to filter which alerts it receives. Conditions are written in the Expr language.

In addition to Expr's built-in functions, Uptrace provides:

Function	Returns
`monitorName() string`	The monitor name.
`alertName() string`	The alert name.
`alertType() string`	The alert kind: `"threshold"`, `"anomaly"`, `"outlier"`, `"forecast"`, `"error"`, or `"slo"`.
`attr(key string) string`	The value of an alert attribute.
`hasAttr(key string) bool`	`true` if the attribute exists on the alert.

Route only prod alerts to a Slack channel:

text

attr("deployment_environment") == "prod"

Route only prod hosts to PagerDuty:

text

attr("host_name") startsWith "prod-"

Route only URGENT monitors:

text

monitorName() contains "URGENT"

Route threshold alerts to one channel, error alerts to another:

text

alertType() == "threshold"

Use alertType() == "error" on the error channel. To include every metric detector, match threshold, anomaly, outlier, and forecast.

Notification frequency

On the first occurrence, Uptrace creates an alert and sends a notification. The default repeat interval increases over time to reduce alert fatigue. A monitor can use a different repeat strategy.

Metric and SLO monitors — the default interval starts at 15 minutes and doubles every 3 notifications. Maximum is 24 hours.

Notifications sent	Interval
1–3	15 minutes
4–6	30 minutes
7–9	1 hour
10–12	2 hours
...	doubles every 3
max	24 hours

Error monitors — the default interval starts at 1 hour and doubles every 2 notifications. Maximum is 1 week.

Notifications sent	Interval
1–2	1 hour
3–4	2 hours
5–6	4 hours
7–8	8 hours
...	doubles every 2
max	1 week

When a metric or SLO alert resolves, Uptrace sends a recovery notification. Error alerts do not resolve automatically. With the default repeat strategy, an open metric or SLO alert can notify every 24 hours indefinitely.

Email notifications

Relevant for the self-hosted version only.

To receive email notifications in Uptrace Community, users must have correct email addresses configured and the mailer option must be enabled in the Uptrace config file.