Uptrace: Querying Spans

Introduction

Uptrace provides a powerful querying language that supports filters (where _status_code = "error), grouping (group by _group_id), and aggregates (p50(_duration)).

Filters

To write useful and performant queries, you need to pre-process raw data so it has a well-defined structure. You can achieve that by recording contextual information in span attributesopen in new window and eventsopen in new window. For logs, you can use structured loggingopen in new window.

Identifiers

Identifiers are unquoted strings, such as _name, display_name, _duration, etc. You can use them to reference span fields, attributes, and extract values from JSON.

Span fields start with an underscore so they don't conflict with attributes.

Span fieldDescription
_idSpan id.
_parent_idSpan parent id.
_trace_idTrace id.
_nameSpan name.
_event_nameEvent name.
_kindSpan kind
_durationSpan duration in microseconds.
_timeSpan time.
_status_codeSpan status code.
_status_messageSpan status message.

Attribute names are mostly unchanged except that dots are replaced by underscores, for example:

Attribute nameDescription
display_namedisplay.name attribute.
service_nameOpenTelemetry service.name attribute.

Strings

Strings can be single or double quoted, for example:

"I'm a string\n"
'I\'m a string\n'

You can also use backticks to define strings that don't require/support any escape sequences, for example, to work with regular expressions:

replace_regexp(host_name, `^some-prefix-(\w+)$`, `\1`)

Filters

Uptrace allows to filter spans and events by their attributes. Filters start with the keyword where, for example, where display_name contains 'hello' or _count > 100.

To filter query results, replace where prefix with having, for example, having p50(_duration) > 100ms.

Uptrace supports the following span attribute types:

Attribute typeSupported comparison operators
string=, in, like, contains, ~ (regexp), exists
int64 and float64=, <, <=, >, >=, exists
bool=, !=
string arraycontains, exists
Uptrace filterDescription
where _status_code = "error"Filter spans with error status code. Case-sensitive.
where display_name like "hello%"Filter span names that start with "hello". Case-insensitive.
where display_name like "%hello"Filter span names that end with "hello". Case-insensitive.
where display_name contains "hello"Filter span names that contain "hello". Case-insensitive.
where display_name contains "foo|bar"Same as display_name contains "foo" OR display_name contains "bar".
where _duration > 1msSame as _duration > 1000. Uptrace supports μs, ms, and s units.
where http_request_content_length > 1kbSame as http_request_content_length > 1024. Uptrace supports kb, mb, gb, and tb units.
where _event_count > 0Filter spans with events.
where _event_error_count > 0Filter spans with error events.
where _event_log_count > 0Filter spans with log events.
where foo existsFilter spans that have attribute foo.

Grouping

Grouping expressions start with group by and work just like the corresponding SQL clause, for example, group by host_name groups spans by the attribute host_name and at the same time selects the host_name.

Uptrace groupingNote
group by _group_idGroups similar spans together.
group by _start_of_minuteGroups spans by the minute they were created. Uptrace also supports grouping by hour, day, and week.
group by host_nameGroups spans by the host_name attribute.
group by service_name, service_versionGroups spans by the combination of service_name and service_version attributes.
group by lower(attribute)Lowers the case of the attribute value.

Aggregate functions

Aggregate functions perform a calculation on a set of values, and return a single value. They are often used together with grouping.

Aggregate functionExampleNote
countcount()Number of matched spans/logs/events.
anyany(_name)Any (random) span name.
anyLastanyLast(_name)Any last span name.
avgavg(_duration)Average span duration.
min, maxmax(_duration)Maximum span duration.
sumsum(http_request_content_length)Total number of processed bytes.
p50, p75, p90, p99p50(_duration)Span duration percentile.
top3, top10top3(code_function)Top 3 most popular function names.
uniquniq(http_client_ip)Number of unique IP addresses.

Uptrace also supports ClickHouse ifopen in new window combinator on aggregate functions, for example:

  • countIf(_status_code = "error"). Number of matched spans with _status_code = "error".
  • p50If(_duration, service_name = "service1"). P50 duration for the service service1.

Uptrace also provides shortcuts for common aggregations:

Virtual columnNote
_error_rateAn alis for countIf(_status_code = "error") / count().

Transform functions

Transform functions accept a value and return a new value for each matched span/log/event.

FunctionExampleNote
lowerlower(log_severity)Lowers the string case.
upperlower(log_severity)Lowers the string case.
perMinperMin(sum(_count))Divides the value by the number of minutes in the interval.
perSecperSec(sum(_count))Divides the value by the number of seconds in the interval.
extract(haystack, pattern)extract(host_name, `^uptrace-prod-(\w+)$`)Extracts a fragment of the haystack string using the regular expression pattern.
replace(haystack, substring, replacement)replace(host_name, 'uptrace-prod-', '')Replaces all occurrences of the substring in haystack by the replacement string.
replaceRegexp(haystack, pattern, replacement)replace_regexp(host, `^`, 'prefix ')Replaces all occurrences of the substring matching the regular expression pattern in haystack by the replacement string.
arrayJoinarrayJoin(db_sql_tables)See ClickHouse arrayJoinopen in new window.
parseDateTimeparseDateTime(str_with_time)Parses a string as a date with time.
toStartOfDay*toStartOfDay(_time)Rounds down a time to the start of the day.

In addition to toStartOfDay, Uptrace also supports toStartOfHour, toStartOfMinute, toStartOfSecond, toStartOfFiveMinutes, toStartOfTenMinutes, and toStartOfFifteenMinutes.

Combining all together

You can write powerful queries combining filters, grouping, and aggregates together. For example, to select the number of unique visitors for each day excluding bots:

where user_agent_is_bot not exists | uniq(client_address) | group by toStartOfDay(_time)

Querying

Last Updated: