Uptrace: Querying Spans


Uptrace provides a powerful querying language that supports filters (where _status_code = "error), grouping (group by _group_id), and aggregates (p50(_duration)).


To write useful and performant queries, you need to pre-process raw data so it has a well-defined structure. You can achieve that by recording contextual information in span attributesopen in new window and eventsopen in new window. For logs, you can use structured loggingopen in new window.


Uptrace allows to filter spans and events by their attributes. Filters start with the keyword where, for example, where .name contains 'hello' or .count > 100. Uptrace automatically translates filters into SQL WHERE or HAVING, so you don't have to worry about that.

Uptrace supports the following span attribute types:

Attribute typeSupported comparison operators
string=, like, contains, ~ (regexp), exists
int64 and float64=, <, <=, >, >=, exists
string arraycontains, exists
Uptrace filterDescription
where _status_code = "error"Filter spans with error status code. Case-sensitive.
where display_name like "hello%"Filter span names that start with "hello". Case-insensitive.
where display_name like "%hello"Filter span names that end with "hello". Case-insensitive.
where display_name contains "hello"Filter span names that contain "hello". Case-insensitive.
where display_name contains "foo|bar"Same as .name contains "foo" OR .name contains "bar".
where _duration > 1msSame as _duration > 1000. Uptrace supports μs, ms, and s units.
where http_request_content_length > 1kbSame as http.request_content_length > 1024. Uptrace supports kb, mb, gb, and tb units.
where _event_count > 0Filter spans with events.
where _event_error_count > 0Filter spans with error events.
where _event_log_count > 0Filter spans with log events.
where _is_eventFilter event spans, for example, exceptions or logs.
where foo existsFilter spans that have attribute foo.


Grouping expressions start with group by and work just like the corresponding SQL clause, for example, group by host.name groups spans by the attribute host.name and at the same time selects the host.name.

Uptrace groupingNote
group by _group_idGroup similar spans together.
group by _start_of_minuteGroup spans by the minute they were created. Uptrace also supports grouping by hour, day, and week.
group by host_nameGroup spans by the host.name attribute.
group by service_name, service_versionGroup spans by the combination of service.name and service.version attributes.


Aggregate functions perform a calculation on a set of values, and return a single value. They are often used together with grouping.

Aggregate functionExampleNote
anyany(_name)Any (random) span name.
avgavg(_duration)Average span duration.
min, maxmax(_duration)Maximum span duration.
p50, p75, p90, p99p50(_duration)Span duration percentile.
sumsum(http_request_content_length)Total number of processed bytes.
top3, top10top3(code_function)Top 3 most popular function names.
uniquniq(http_client_ip)Number of unique IP addresses.

There is also a number of common pre-aggregated columns:

Virtual columnNote
_countThe equivalent of SQL count(*) that takes in account adjusted countsopen in new window.
_error_countThe number of spans with _status.code = 'error'.
_error_rateThe result of _error_count / _count.

Combining all together

You can write powerful queries combining filters, grouping, and aggregates together. For example, to select the number of unique visitors for each day excluding bots:

where user_agent_is_bot not exists | uniq(client_address) | group by _start_of_day


Last Updated: