Uptrace: Querying Spans
Introduction
Uptrace provides a powerful querying language that supports filters (where span.status.code = "error), grouping (group by span.group_id), and aggregates (p50(span.duration)).
To write useful and performant queries, you need to pre-process raw data so it has a well-defined structure. You can achieve that by recording contextual information in span attributes and events. For logs, you can use structured logging.
Filters
Uptrace allows to filter spans and events by their attributes. Filters start with the keyword where, for example, where span.name contains 'hello' or span.count > 100. Uptrace automatically translates filters into SQL WHERE or HAVING, so you don't have to worry about that.
Uptrace supports the following span attribute types:
| Attribute type | Supported comparison operators |
|---|---|
string | =, like, contains, exists |
int64 and float64 | =, <, <=, >, >=, exists |
array of the types above | contains, exists |
| Uptrace filter | Description |
|---|---|
span.status_code = "error" | Filter spans with error status code. Case-sensitive. |
span.name like "hello%" | Filter span names that start with "hello". Case-insensitive. |
span.name like "%hello" | Filter span names that end with "hello". Case-insensitive. |
span.name contains "hello" | Filter span names that contain "hello". Case-insensitive. |
span.name contains "foo|bar" | Same as span.name contains "foo" OR span.name contains "bar". |
span.duration > 1ms | Same as span.duration > 1000. Uptrace supports μs, ms, and s units. |
http.request_content_length > 1kb | Same as http.request_content_length > 1024. Uptrace supports kb, mb, gb, and tb units. |
span.event_count > 0 | Filter spans with events. |
span.event_error_count > 0 | Filter spans with error events. |
span.event_log_count > 0 | Filter spans with log events. |
where span.is_event | Filter event spans, for example, exceptions or logs. |
where foo exists | Filter spans that have attribute foo. |
Grouping
Grouping expressions start with group by and work just like the corresponding SQL clause, for example, group by host.name groups spans by the attribute host.name and at the same time selects the host.name.
| Uptrace grouping | Note |
|---|---|
group by span.group_id | Group similar spans together. |
group by span.start_of_minute | Group spans by the minute they were created. Uptrace also supports grouping by hour, day, and week. |
group by host.name | Group spans by the host.name attribute. |
group by service.name, service.version | Group spans by the combination of service.name and service.version attributes. |
Aggregates
Aggregate functions perform a calculation on a set of values, and return a single value. They are often used together with grouping.
| Aggregate function | Example | Note |
|---|---|---|
any | any(span.name) | Any (random) span name. |
avg | avg(span.duration) | Average span duration. |
min, max | max(span.duration) | Maximum span duration. |
p50, p75, p90, p99 | p50(span.duration) | Span duration percentile. |
sum | sum(http.request_content_length) | Total number of processed bytes. |
top3, top10 | top3(code.function) | Top 3 most popular function names. |
uniq | uniq(http.client_ip) | Number of unique IP addresses. |
There is also a number of common pre-aggregated columns:
| Virtual column | Note |
|---|---|
span.count | The equivalent of SQL count(*) that takes in account adjusted counts. |
span.error_count | The number of spans with span.status.code = 'error'. |
span.error_pct | The result of span.error_count / span.count. |
span.count_per_min | The average number of spans per minute. |
Combining all together
You can write powerful queries combining filters, grouping, and aggregates together. For example, to select the number of unique visitors for each day excluding bots:
where http.user_agent.bot not exists | uniq(http.client_ip) | group by span.start_of_day
Nested queries
Uptrace also supports nested queries, for example, the following 2 queries:
# group spans by the group id while counting number of spans
group by span.group_id | span.count as num_span
# calculate average/min/max number of spans in the groups from the previous query
count() | avg(num_span) | min(num_span) | max(num_span)
are compiled to the following pseudo-SQL:
SELECT count(), avg(num_span), min(num_span), max(num_span)
FROM (
SELECT
span.group_id,
count() AS num_span
GROUP BY span.group_id
)
