Converting spans to metrics

Uptrace allows you to create metrics from spans so they can be monitored using alerting rules.

• Spans metric
• Events metric
• Errors and logs monitoring

Spans metric

First, create uptrace.tracing.spans metric from incoming spans:

metrics_from_spans:
  - name: uptrace.tracing.spans
    description: Spans count (excluding events)
    instrument: counter
    unit: 1
    value: span.count
    attrs:
      - span.system as system
      - service.name as service
      - host.name as host
      - span.status_code as status
    where: not span.is_event

Then, monitor that metric using an alerting rule:

alerting:
  rules:
    - name: Service has high error rate
      metrics:
        - uptrace.tracing.spans as $spans
      query:
        - $spans{status="error"} / $spans > 0.1 group by service.name
      for: 5m

See OpenTelemetry Metrics for a list of supported instrument names and more.

Events metric

You can also create a separate metric for events:

metrics_from_spans:
  - name: uptrace.tracing.events
    description: Events count (excluding spans)
    instrument: counter
    unit: 1
    value: span.count
    attrs:
      - span.system as system
      - service.name as service
      - host.name as host
    where: span.is_event

And logs:

metrics_from_spans:
  - name: uptrace.tracing.logs
    description: Logs count
    instrument: counter
    unit: 1
    value: span.count
    attrs:
      - service.name as service
      - host.name as host
      - log.severity as severity
    where: span.system like 'log:%'

Errors and logs monitoring

You can also monitor individual events for a specific message or an attribute.

For example, to create an alert when a log message contains some pattern:

metrics_from_spans:
  - name: uptrace.tracing.suspicious_logs
    description: Number of logs with suspicious log messages
    instrument: counter
    unit: 1
    value: span.count
    attrs:
      - service.name as service
      - host.name as host
      - log.severity as severity
    where: log.message like '%sus%'
    annotations:
      - span.trace_id
      - log.message

You can then use annotations in alerting rules.