Uptrace: Querying Spans


Uptrace provides a powerful querying language that supports filters (where .status.code = "error), grouping (group by .group_id), and aggregates (p50(.duration)).


To write useful and performant queries, you need to pre-process raw data so it has a well-defined structure. You can achieve that by recording contextual information in span attributesopen in new window and eventsopen in new window. For logs, you can use structured loggingopen in new window.


Uptrace allows to filter spans and events by their attributes. Filters start with the keyword where, for example, where .name contains 'hello' or .count > 100. Uptrace automatically translates filters into SQL WHERE or HAVING, so you don't have to worry about that.

Uptrace supports the following span attribute types:

Attribute typeSupported comparison operators
string=, like, contains, ~ (regexp), exists
int64 and float64=, <, <=, >, >=, exists
string arraycontains, exists
Uptrace filterDescription
where .status_code = "error"Filter spans with error status code. Case-sensitive.
where display.name like "hello%"Filter span names that start with "hello". Case-insensitive.
where display.name like "%hello"Filter span names that end with "hello". Case-insensitive.
where display.name contains "hello"Filter span names that contain "hello". Case-insensitive.
where display.name contains "foo|bar"Same as .name contains "foo" OR .name contains "bar".
where .duration > 1msSame as .duration > 1000. Uptrace supports μs, ms, and s units.
where http.request_content_length > 1kbSame as http.request_content_length > 1024. Uptrace supports kb, mb, gb, and tb units.
where .event_count > 0Filter spans with events.
where .event_error_count > 0Filter spans with error events.
where .event_log_count > 0Filter spans with log events.
where .is_eventFilter event spans, for example, exceptions or logs.
where foo existsFilter spans that have attribute foo.


Grouping expressions start with group by and work just like the corresponding SQL clause, for example, group by host.name groups spans by the attribute host.name and at the same time selects the host.name.

Uptrace groupingNote
group by .group_idGroup similar spans together.
group by .start_of_minuteGroup spans by the minute they were created. Uptrace also supports grouping by hour, day, and week.
group by host.nameGroup spans by the host.name attribute.
group by service.name, service.versionGroup spans by the combination of service.name and service.version attributes.


Aggregate functions perform a calculation on a set of values, and return a single value. They are often used together with grouping.

Aggregate functionExampleNote
anyany(.name)Any (random) span name.
avgavg(.duration)Average span duration.
min, maxmax(.duration)Maximum span duration.
p50, p75, p90, p99p50(.duration)Span duration percentile.
sumsum(http.request_content_length)Total number of processed bytes.
top3, top10top3(code.function)Top 3 most popular function names.
uniquniq(http.client_ip)Number of unique IP addresses.

There is also a number of common pre-aggregated columns:

Virtual columnNote
.countThe equivalent of SQL count(*) that takes in account adjusted countsopen in new window.
.error_countThe number of spans with .status.code = 'error'.
.error_rateThe result of .error_count / .count.

Combining all together

You can write powerful queries combining filters, grouping, and aggregates together. For example, to select the number of unique visitors for each day excluding bots:

where http.user_agent.bot not exists | uniq(http.client_ip) | group by .start_of_day


Last Updated: