HomeOpenTelemetryInstrumentationsUptrace DemoGitHub

Alerting and Notifications

Uptrace allows to monitor metrics using alerting rules and can send notifications via email, Slack, and Telegram using AlertManager integration.

Alerting

You can define alerting rules in the Uptrace config file, for example, to alert when there are network errors for the last 5 minutes:

# uptrace.yml

alerting:
  rules:
    - name: Network errors
      metrics:
        - system.network.errors as $net_errors
      query:
        - $net_errors > 0
      # for the last 5 minutes
      for: 5m

You can also create more complex queries, for example, to monitor filesystem utilization on each host name:

# uptrace.yml

alerting:
  rules:
    - name: Filesystem usage >= 90%
      metrics:
        - system.filesystem.usage as $fs_usage
      query:
        - group by host.name
        - group by device
        - where device !~ "loop"
        - $fs_usage{state="used"} / $fs_usage >= 0.9
      for: 5m
      # List of projects to monitor.
      projects: [1, 2]

See Querying Metrics for details about the query language.

AlertManager

AlertManager integration allows to send notifications via email, Slack, Telegram, and others. To configure the integration, add the following section to the Uptrace config:

# uptrace.yml

## AlertManager client configuration.
alertmanager_client:
  # AlertManager API endpoints that Uptrace uses to manage alerts.
  urls:
    - 'http://localhost:9093/api/v2/alerts'

Note that the config above is NOT an AlertManager config and you need to configureopen in new window AlertManager separately.

Email notifications

To configure email notifications, add the following to AlertManager config:

# alertmanager.yml

global:
  # The smarthost and SMTP sender used for mail notifications.
  smtp_smarthost: 'smtp.gmail.com:587'
  smtp_from: '[SENDER]@gmail.com'
  smtp_auth_username: '[SENDER]@gmail.com'
  smtp_auth_identity: '[SENDER]@gmail.com'
  smtp_auth_password: '[APP_PASSWORD]'

Note that Gmail does not allow to use your real password in smtp_auth_password. Intead, you should generate an app password for Gmail:

  1. In Gmail, click on your avatar -> "Manage your Google Account".
  2. On the left, click on "Security".
  3. Scroll to "Signing in to Google" and click on "App password".

See Gmail documentationopen in new window for details.

Slack notifications

To configure Slack notifications, you need to create a Slack webhook and configure AlertManager to use it:

# alertmanager.yml

receivers:
  - name: 'team-X'
    slack_configs:
      - api_url: 'https://hooks.slack.com/services/T00000000/B00000000/XXXXXXXXXXXXXXXXXXXXXXXX'
        channel: '#channel'

Example

See Dockeropen in new window example for a quick way to experiment with notifications.

Templating

You can use Go templates to customize rule annotations, for example:

# uptrace.yml

alerting:
  rules:
    - name: Network errors
      metrics:
        - system.network.errors as $net_errors
      query:
        - $net_errors > 0 group by host.name
      for: 5m
      annotations:
        summary: 'Host {{ $labels.host_name }} has network errors'
        description:
          '{{ $labels.host_name }} has high number of net errors: {{ $values.net_errors }}'

In templates, you can use sprigopen in new window functions and have access to the following variables:

  • $labels map[string]string - timeseries attributes.
  • $values map[string]float64 - metric values.
  • $annotations map[string]string - existing annotations coming from the metrics.

You can then use the annotations in AlertManager templates to customize the text sent via Slack, for example:

# alertmanager.yml

- name: 'team-x'
  slack_configs:
    - channel: '#alerts'
      # Alertmanager templates apply here.
      text:
        "<!channel> \nsummary: {{ .CommonAnnotations.summary }}\ndescription: {{
        .CommonAnnotations.description }}"
Last Updated: