Uptrace: Querying Spans

• Introduction
• Identifiers
• Strings
• Filters
• Grouping
• Aggregates
• Combining all together

Introduction

Uptrace provides a powerful querying language that supports filters (where _status_code = "error), grouping (group by _group_id), and aggregates (p50(_duration)).

To write useful and performant queries, you need to pre-process raw data so it has a well-defined structure. You can achieve that by recording contextual information in span attributes and events. For logs, you can use structured logging.

Identifiers

Identifiers are unquoted strings, such as _name, display_name, _duration, etc. You can use them to reference span fields, attributes, and extract values from JSON.

Span fields start with an underscore so they don't conflict with attributes.

Span field	Description
_id	Span id.
_parent_id	Span parent id.
_trace_id	Trace id.
_name	Span name.
_event_name	Event name.
_kind	Span kind
_duration	Span duration in microseconds.
_time	Span time.
_status_code	Span status code.
_status_message	Span status message.

Attribute names are mostly unchanged except that dots are replaced by underscores, for example:

Attribute name	Description
display_name	display.name attribute.
service_name	OpenTelemetry service.name attribute.

You can use dot notation to access sub-fields in nested structures such as JSON, for example, data.key1.subkey1 can extract the field value from the following JSON:

{
  "data": {
    "key1": {
      "subkey1": "value1"
    }
  }
}

The JSON path can be a maximum of 3 levels deep, for example, one.two.three is allowed, but one.two.three.four won't find anything. Arrays are not supported.

Strings

Strings can be single or double quoted, for example:

"I'm a string\n"
'I\'m a string\n'

You can also use backticks to define strings that don't require/support any escape sequences, for example, to work with regular expressions:

replace_regexp(host_name, `^some-prefix-(\w+)$`, `\1`)

Filters

Uptrace allows to filter spans and events by their attributes. Filters start with the keyword where, for example, where display_name contains 'hello' or _count > 100. Uptrace automatically translates filters into SQL WHERE or HAVING, so you don't have to worry about that.

Uptrace supports the following span attribute types:

Attribute type	Supported comparison operators
string	=, in, like, contains, ~ (regexp), exists
int64 and float64	=, <, <=, >, >=, exists
string array	contains, exists

Uptrace filter	Description
where _status_code = "error"	Filter spans with error status code. Case-sensitive.
where display_name like "hello%"	Filter span names that start with "hello". Case-insensitive.
where display_name like "%hello"	Filter span names that end with "hello". Case-insensitive.
where display_name contains "hello"	Filter span names that contain "hello". Case-insensitive.
where display_name contains "foo|bar"	Same as display_name contains "foo" OR display_name contains "bar".
where _duration > 1ms	Same as _duration > 1000. Uptrace supports μs, ms, and s units.
where http_request_content_length > 1kb	Same as http_request_content_length > 1024. Uptrace supports kb, mb, gb, and tb units.
where _event_count > 0	Filter spans with events.
where _event_error_count > 0	Filter spans with error events.
where _event_log_count > 0	Filter spans with log events.
where _is_event	Filter event spans, for example, exceptions or logs.
where foo exists	Filter spans that have attribute foo.

Grouping

Grouping expressions start with group by and work just like the corresponding SQL clause, for example, group by host_name groups spans by the attribute host_name and at the same time selects the host_name.

Uptrace grouping	Note
group by _group_id	Groups similar spans together.
group by _start_of_minute	Groups spans by the minute they were created. Uptrace also supports grouping by hour, day, and week.
group by host_name	Groups spans by the host_name attribute.
group by service_name, service_version	Groups spans by the combination of service_name and service_version attributes.
group by lower(attribute)	Lowers the case of the attribute value.
group by upper(attribute)	Uppers the case of the attribute value.
group by extract(haystack, pattern)	Extracts a fragment of the haystack string using the regular expression pattern.
group by replace(haystack, substring, replacement)	Replaces all occurrences of the substring in haystack by the replacement string.
group by replace_regexp(haystack, pattern, replacement)	Replaces all occurrences of the substring matching the regular expression pattern in haystack by the replacement string.

Aggregates

Aggregate functions perform a calculation on a set of values, and return a single value. They are often used together with grouping.

Aggregate function	Example	Note
any	any(_name)	Any (random) span name.
avg	avg(_duration)	Average span duration.
min, max	max(_duration)	Maximum span duration.
p50, p75, p90, p99	p50(_duration)	Span duration percentile.
sum	sum(http_request_content_length)	Total number of processed bytes.
top3, top10	top3(code_function)	Top 3 most popular function names.
uniq	uniq(http_client_ip)	Number of unique IP addresses.

There is also a number of common pre-aggregated columns:

Virtual column	Note
_count	The equivalent of SQL count(*) that takes in account adjusted counts.
_error_count	The number of spans with _status_code = 'error'.
_error_rate	The result of _error_count / _count.

Combining all together

You can write powerful queries combining filters, grouping, and aggregates together. For example, to select the number of unique visitors for each day excluding bots:

where user_agent_is_bot not exists | uniq(client_address) | group by _start_of_day