Querying Traces

While the span query language operates on individual spans and logs, trace queries allow you to select entire traces based on the spans, logs, and events they contain. This is useful for finding traces that match complex criteria across multiple services and operations.

For example, you can find all traces that contain a slow database query, have an error in a specific service, or involve a particular combination of microservices.

Query Structure

A trace query consists of multiple rows, where each row targets a different part of the trace. The first row is always the root row, and subsequent rows define conditions on child spans, logs, or events.

root: <query for the root span>
<system> as <alias>: <query for matching spans>

Each row accepts the full span query language including filters, aggregations, and groupings. A typical root row query looks like:

root: perMin(count()) | quantiles(_dur_ms) | group by _group_id

Filtering by System

Each non-root row targets spans belonging to a specific system. You can use exact systems or wildcard systems with :all:

Examples

Traces with Multiple Database Queries

Find traces that contain at least 2 PostgreSQL queries:

root: <empty>
db:postgresql as pg: having count() >= 2

This is useful for identifying N+1 query problems or traces with excessive database calls.

Traces with Slow Service Calls

Find traces where a specific service takes longer than 1 second to respond:

root: <empty>
rpc:all as rpc: where service_name = "foo" | where _kind = "server" | where _dur_ms >= 1000

Traces with Error Logs

Find traces that contain error logs mentioning "timeout":

root: <empty>
log:error as err: _display_name contains "timeout"

Traces with Errors in a Specific Service

Find traces where a particular service returned an error:

root: <empty>
rpc:all as rpc: where service_name = "payment-service" | where _status_code = "error"

Traces with Slow Root Spans

Find traces where the root span exceeds 5 seconds:

root: where _dur_ms >= 5000

Combining Multiple Conditions

Find traces that are slow and contain database errors:

root: where _dur_ms >= 3000
db:all as db: where _status_code = "error"

This selects traces where the overall duration exceeds 3 seconds and at least one database span has an error status.

Available Clauses

Each row — both root and child — supports the full span query language including filters, aggregations, and groupings:

Multiple clauses can be chained with the pipe | operator:

rpc:all as rpc: where service_name = "foo" | where _kind = "server" | where _dur_ms >= 1000

See Also

• Querying spans and logs - Query language fundamentals
• Searching spans and logs - Simpler search syntax
• Grouping and systems - How span systems work