Alerting and Notifications
Uptrace allows to monitor metrics using alerting rules and can send notifications via email, Slack, and Telegram using AlertManager integration.
Alerting
You can define alerting rules in the Uptrace config file, for example, to alert when there are network errors for the last 5 minutes:
# uptrace.yml
alerting:
rules:
- name: Network errors
metrics:
- system.network.errors as $net_errors
query:
- $net_errors > 0
# for the last 5 minutes
for: 5m
You can also create more complex queries, for example, to monitor filesystem utilization on each host name:
# uptrace.yml
alerting:
rules:
- name: Filesystem usage >= 90%
metrics:
- system.filesystem.usage as $fs_usage
query:
- group by host.name
- group by device
- where device !~ "loop"
- $fs_usage{state="used"} / $fs_usage >= 0.9
for: 5m
# List of projects to monitor.
projects: [1, 2]
See Querying Metrics for details about the query language.
AlertManager
AlertManager integration allows to send notifications via email, Slack, Telegram, and others. To configure the integration, add the following section to the Uptrace config:
# uptrace.yml
## AlertManager client configuration.
alertmanager_client:
# AlertManager API endpoints that Uptrace uses to manage alerts.
urls:
- 'http://localhost:9093/api/v2/alerts'
Note that the config above is NOT an AlertManager config and you need to configure AlertManager separately.
Email notifications
To configure email notifications, add the following to AlertManager config:
# alertmanager.yml
global:
# The smarthost and SMTP sender used for mail notifications.
smtp_smarthost: 'smtp.gmail.com:587'
smtp_from: '[SENDER]@gmail.com'
smtp_auth_username: '[SENDER]@gmail.com'
smtp_auth_identity: '[SENDER]@gmail.com'
smtp_auth_password: '[APP_PASSWORD]'
Note that Gmail does not allow to use your real password in smtp_auth_password
. Intead, you should generate an app password for Gmail:
- In Gmail, click on your avatar -> "Manage your Google Account".
- On the left, click on "Security".
- Scroll to "Signing in to Google" and click on "App password".
See Gmail documentation for details.
Slack notifications
To configure Slack notifications, you need to create a Slack webhook and configure AlertManager to use it:
# alertmanager.yml
receivers:
- name: 'team-X'
slack_configs:
- api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX'
channel: '#channel'
Example
See Docker example for a quick way to experiment with notifications.
Templating
You can use Go templates to customize rule annotations, for example:
# uptrace.yml
alerting:
rules:
- name: Network errors
metrics:
- system.network.errors as $net_errors
query:
- $net_errors > 0 group by host.name
for: 5m
annotations:
summary: 'Host {{ $labels.host_name }} has network errors'
description:
'{{ $labels.host_name }} has high number of net errors: {{ $values.net_errors }}'
In templates, you can use sprig functions and have access to the following variables:
$labels map[string]string
- timeseries attributes.$values map[string]float64
- metric values.$annotations map[string]string
- existing annotations coming from the metrics.
You can then use the annotations in AlertManager templates to customize the text sent via Slack, for example:
# alertmanager.yml
- name: 'team-x'
slack_configs:
- channel: '#alerts'
# Alertmanager templates apply here.
text:
"<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{
.CommonAnnotations.description }}"