Grouping similar spans and events together

Uptrace automatically groups similar spans and events together to help you identify patterns and anomalies in your application. This page explains how grouping works, how to optimize span names for effective grouping, and how to use OpenTelemetry semantic conventions for different types of operations.

How grouping works

Uptrace computes a hash for each span based on its type and key attributes. Spans with the same hash are grouped together. The hash always includes:

• Project ID - spans from different projects are never grouped together
• System - the detected span type (e.g., http, db:postgresql, messaging:rabbitmq)
• Span name (for spans) or Event name (for events)
• Span kind (for spans) - client, server, producer, consumer, or internal

Additionally, each span type includes type-specific attributes in the hash:

Uptrace supports both stable and legacy OpenTelemetry semantic convention attribute names. For example, both db.system.name (stable) and db.system (legacy) are recognized.

Span names

Uptrace uses span names and some attributes to group similar spans together. To group spans properly, give them short and concise names. The total number of unique span names should be less than 1000. Otherwise, you will have too many span groups and your experience may suffer.

The following names are good because they are short, distinctive, and help grouping similar spans together:

The following names are bad because they contain variable params and args:

Display name

Because OpenTelemetry uses span and event names to group similar spans together, such names end up being not very descriptive, for example, SQL spans often have names like SELECT or INSERT.

This is where the display.name attribute comes in handy. The display.name is a human-readable string that provides a short summary of the operation that the span or event represents.

Uptrace doesn't use display names for grouping so they don't have the same limitations as span names. For example, a span representing a SQL query might have the following name and attributes:

SpanName = "SELECT"

display.name = "pg_select_items_from_group(123)"
db.system.name = "postgresql"
db.query.text = "SELECT * FROM items WHERE group_id = 123 ORDER BY id LIMIT 100"

You can use display.name attribute with events too.

Custom grouping

You can customize how Uptrace groups spans and events together by specifying the grouping.fingerprint attribute which can be a string or a number (hash). Uptrace will group spans/events with the same fingerprint together.

SpanName = "SELECT"

display.name = "pg_select_items_from_group(123)"
db.system.name = "postgresql"
db.query.text = "SELECT * FROM items WHERE group_id = 123 ORDER BY id LIMIT 100"
grouping.fingerprint = "select group items"

Grouping settings

On the Project Settings page, there are several options that control how Uptrace groups spans together:

• Group by environment - Group spans with different deployment.environment.name (or legacy deployment.environment) attribute separately. After enabling this setting, there will be separate groups for spans with the same name but different environments.
• Group functions by service - Group spans with funcs system and different service.name attribute separately. After enabling this setting, there will be a separate system for each service such as funcs:service1 and funcs:service2.

Span systems

Depending on the presence of some semantic attributes, Uptrace assigns each span a system, for example:

• http:service_name system for HTTP spans.
• db:postgresql for PostgreSQL queries.
• log:error for log messages with ERROR severity.
• exceptions for exceptions.

Using a system, you can easily filter spans that have the same set of attributes, for example, all database spans.

HTTP

To monitor HTTP clients and servers, use the HTTP semantic conventions.

A minimal HTTP server example:

SpanName = "GET /users/:id"
SpanKind = "server"

http.request.method = "GET"
http.route = "/users/:id"
url.scheme = "https"

A minimal HTTP client example:

SpanName = "GET"
SpanKind = "client"

http.request.method = "GET"
url.full = "https://api.example.com/users/123"
server.address = "api.example.com"

RPC

To monitor remote procedure calls, use the RPC semantic conventions.

A minimal RPC server example:

SpanName = "AuthService/Auth"
SpanKind = "server"

rpc.system = "grpc"
rpc.service = "AuthService"
rpc.method = "Auth"

Database

To monitor database queries and Redis/memcached commands, use the database semantic conventions.

A minimal DB example:

SpanName = "SELECT users"
SpanKind = "client"

db.system.name = "postgresql"
db.query.text = "SELECT * FROM users WHERE id = $1"
db.collection.name = "users"
db.namespace = "mydb"

A minimal Redis command example:

SpanName = "GET"
SpanKind = "client"

db.system.name = "redis"
db.query.text = "GET foo"

Messages

To monitor producers and consumers (queues), use the messaging semantic conventions.

A minimal producer example:

SpanName = "MyQueue send"
SpanKind = "producer"

messaging.system = "rabbitmq"
messaging.destination.name = "MyQueue"
messaging.operation.type = "send"

A minimal consumer example:

SpanName = "MyQueue process"
SpanKind = "consumer"

messaging.system = "rabbitmq"
messaging.destination.name = "MyQueue"
messaging.operation.type = "process"

Functions

To monitor functions, use source code attributes.

A minimal example:

SpanName = "org.FetchUser"

service.name = "myservice"
code.function.name = "FetchUser"
code.namespace = "org"
code.filepath = "org/user.go"
code.lineno = 123

Functions as a Service

To monitor serverless functions, use the FaaS semantic conventions.

A minimal HTTP-triggered function:

SpanName = "my-lambda-function"
SpanKind = "server"

faas.trigger = "http"
faas.name = "my-lambda-function"

A datasource-triggered function (e.g., S3, DynamoDB):

SpanName = "my-lambda-function"
SpanKind = "server"

faas.trigger = "datasource"
faas.name = "my-lambda-function"
faas.document.collection = "my-bucket"
faas.document.operation = "insert"

A minimal client example (invoking a function):

SpanName = "my-lambda-function"
SpanKind = "client"

faas.invoked_name = "my-lambda-function"
faas.invoked_provider = "aws"
faas.invoked_region = "us-west-2"

Exceptions

To monitor errors and exceptions, use the exceptions semantic conventions and events API.

A minimal example:

EventName = "exception"

exception.type = "*exec.ExitError"
exception.message = "exit status 1"
exception.stacktrace = "<exception stacktrace>"

span := trace.SpanFromContext(ctx)

span.RecordError(err, trace.WithAttributes(
    attribute.String("exception.type", fmt.Sprintf("%T", err)),
))
// Or manually:
span.AddEvent("exception", trace.WithAttributes(
    attribute.String("exception.type", "*exec.ExitError"),
    attribute.String("exception.message", "exit status 1"),
    attribute.String("exception.stacktrace", string(debug.Stack())),
))

from opentelemetry import trace

span = trace.get_current_span()
try:
    # your code
    pass
except Exception as e:
    span.record_exception(e)
    span.set_status(trace.Status(trace.StatusCode.ERROR, str(e)))

const span = trace.getActiveSpan();
try {
  // your code
} catch (error) {
  span.recordException(error);
  span.setStatus({ code: SpanStatusCode.ERROR, message: error.message });
}

You can also control how Uptrace groups exceptions together by specifying the grouping.fingerprint attribute which can be a string or a number (hash). Uptrace will group exceptions with the same fingerprint together.

EventName = "exception"

exception.type = "*exec.ExitError"
exception.message = "exit status 1"
grouping.fingerprint = "*exec.ExitError"

Logs

Uptrace groups logs by log.severity and log.message_format. The log.message_format attribute should contain the log message template (with placeholders), not the actual message with interpolated values. This ensures logs with the same pattern are grouped together.

A minimal example:

EventName = "log"

log.severity = "info"
log.message = "user 123 logged in from 192.168.1.1"
log.message_format = "user %d logged in from %s"
log.params.user_id = "123"
log.params.ip = "192.168.1.1"

span := trace.SpanFromContext(ctx)

span.AddEvent("log", trace.WithAttributes(
    attribute.String("log.severity", "info"),
    attribute.String("log.message", fmt.Sprintf("user %d logged in from %s", userID, ip)),
    attribute.String("log.message_format", "user %d logged in from %s"),
    attribute.Int("log.params.user_id", userID),
    attribute.String("log.params.ip", ip),
))

from opentelemetry import trace

span = trace.get_current_span()
span.add_event("log", attributes={
    "log.severity": "info",
    "log.message": f"user {user_id} logged in from {ip}",
    "log.message_format": "user %d logged in from %s",
    "log.params.user_id": user_id,
    "log.params.ip": ip,
})

const span = trace.getActiveSpan();
span.addEvent('log', {
  'log.severity': 'info',
  'log.message': `user ${userId} logged in from ${ip}`,
  'log.message_format': 'user %d logged in from %s',
  'log.params.user_id': userId,
  'log.params.ip': ip,
});

You can also control how Uptrace groups logs together by providing grouping.fingerprint attribute which can be a string or a number (hash). This is useful for unstructured logs where you can't easily extract a message format:

EventName = "log"

log.severity = "info"
log.message = "Connection to database failed after 3 retries"
grouping.fingerprint = "database connection failed"

See Monitoring Logs for details.