Grouping similar spans and events together

Span names

Uptrace uses span names and some attributes to group similar spans together. To group spans properly, give them short and concise names. The total number of unique span names should be less than 1000. Otherwise, you will have too many span groups and your experience may suffer.

The following names are good because they are short, distinctive, and help grouping similar spans together:

Span nameComment
GET /projects/:idGood. A route name with param names.
select_projectGood. A function name without arguments.
SELECT * FROM projects WHERE id = ?Good. A database query with placeholders.

The following names are bad because they contain variable params and args:

Span nameComment
GET /projects/42Bad. Contains a variable param 42.
select_project(42)Bad. Contains a variable 42.
SELECT * FROM projects WHERE id = 42Bad. Contains a variable arg 42.

Display name

Because OpenTelemetry uses span and event names to group similar spans together, such names end up not being very descriptive, for example, SQL spans often have names like SELECT or INSERT.

This is where display.name attribute comes in handy. The display name of a span is a human-readable string that provides a short summary of the operation that the span represents.

Backends don't use display names for grouping so they don't have the same limitations as span names. For example, a span representing a SQL query might have the following name and attributes:

SpanName = "SELECT"

display.name = "pg_select_items_from_group(123)"
db.system = "postgresql"
db.statement = "SELECT * FROM items WHERE group_id = 123 ORDER BY id LIMIT 100"

You can use display.name attribute with events too.

Span systems

Depending on the presence of some semantic attributesopen in new window, Uptrace assigns each span a system, for example:

  • http:service_name system for HTTP spans.
  • db:postgresql for PostgreSQL queries.
  • log:error for log messages with ERROR severity.
  • exceptions for exceptions.

Using a system, you can easily filter spans that have the same set of attributes, for example, all database spansopen in new window.

HTTP

To monitor HTTP clients and servers, use HTTP semantic conventionopen in new window.

A minimal HTTP server example:

SpanName = "GET /users/:id"
SpanKind = "server"

http.method = "GET"
http.route = "/users/:id"

A minimal HTTP client example:

SpanName = "GET /users/:id"
SpanKind = "client"

http.method = "GET"
http.route = "/users/:id"

RPC

To monitor remote procedure calls, use RPC semantic conventionopen in new window.

A minimal RPC server example:

SpanName = "AuthService/Auth"
SpanKind = "server"

rpc.system = "grpc"
rpc.service = "AuthService.Auth"
rpc.method = "Auth"

Database

To monitor database queries and Redis/memcached commands, use DB semantic conventionopen in new window.

A minimal DB example:

SpanName = "pg.query"
SpanKind = "client"

db.system = "postgresql"
db.statement = "SELECT * FROM users WHERE id = 123"

A minimal Redis command example:

SpanName = "GET"
SpanKind = "client"

db.system = "redis"
db.statement = "GET foo"

Messages

To monitor producers and consumers (queues), use Messaging semantic conventionopen in new window.

A minimal producer example:

SpanName = "send MyQueue"
SpanKind = "producer"

messaging.system = "rabbitmq"
messaging.destination.name = "MyQueue"
messaging.destination.kind = "queue"

A minimal consumer example:

SpanName = "process MyQueue"
SpanKind = "consumer"

messaging.system = "rabbitmq"
messaging.operation = "process"
messaging.destination.name = "MyQueue"
messaging.destination.kind = "queue"

Functions

To monitor functions, use Source code attributesopen in new window.

A minimal example:

SpanName = "org.FetchUser"

service.name = "myservice"
code.function = "org.FetchUser"
code.filepath = "org/user.go"
code.lineno = "123"

Functions as a Service

To monitor serverless functions, use FaaS semantic conventionopen in new window.

A minimal server example:

SpanName = "my-lambda-function"
SpanKind = "server"

faas.trigger = "http"
faas.name = "my-lambda-function"

A minimal client example:

SpanName = "my-lambda-function"
SpanKind = "client"

faas.trigger = "http"
faas.invoked_name = "my-lambda-function"

Exceptions

To monitor errors and exceptions, use Exceptions semantic conventionopen in new window and events API.

A minimal example:

EventName = "exception"

exception.type = "*exec.ExitError"
exception.message = "exit status 1"
exception.stack = "<exception stack>"

The same in Go programming language:

span := trace.SpanFromContext(ctx)

span.AddEvent("exception", trace.WithAttributes(
    attribute.String("exception.type", "*exec.ExitError"),
    attribute.String("exception.message", "exit status 1"),
    attribute.String("exception.stack", string(runtime.Stack())),
))

You can also control how Uptrace groups exceptions together by providing exception.fingerprint attribute which can be a string or a number (hash):

EventName = "exception"

exception.type = "*exec.ExitError"
exception.message = "exit status 1"
exception.fingerprint = "*exec.ExitError"

Logs

A minimal example:

EventName = "log"

log.severity = "info"
log.message = "something failed"
log.params.key1 = "value1"
log.params.key2 = "value2"

The same in Go programming language:

span := trace.SpanFromContext(ctx)

span.AddEvent("log", trace.WithAttributes(
	attribute.String("log.severity", "info"),
	attribute.String("log.message", "something failed"),
	attribute.String("log.params.key1", "value1"),
	attribute.String("log.params.key2", "value2"),
))

You can also control how Uptrace groups logs together by providing log.fingerprint attribute which can be a string or a number (hash):

EventName = "log"

log.severity = "info"
log.message = "unstructured log message 123 456 789"
log.fingerprint = "unstructured log message"

See Structured logging for more details.

Last Updated: