Metrics query language reference

This page lists every function, operator, and selector the Uptrace metrics query language accepts. To learn how to write queries, start with Querying metrics; to adapt existing Prometheus queries, see PromQL compatibility.

Selectors

A query references a metric through an alias that starts with $:

metrics:
  - system_filesystem_usage as $fs_usage
query:
  - sum($fs_usage)

Attribute filters

Filter a selector by attributes inside curly braces:

You can also filter every expression in a query at once with where, which supports =, !=, <, <=, >, >=, ~, !~, like, not like, in, and not in:

$hits | $misses | where host_name = 'localhost'

Value filter

The _value pseudo-attribute filters timeseries by the datapoint value, not by an attribute. Use it to count series in a given state:

uniq($status{_value=1}) as num_up | uniq($status{_value=0}) as num_down

Time qualifier

$metric._time reads the datapoint timestamp instead of the value. It accepts only min and max, which return the first and the last time the metric reported:

min($cache._time) | max($cache._time)

Offset

offset shifts the query window. A negative offset looks ahead of the evaluation time:

$http_requests_total offset 5m
$http_requests_total offset -5m

Lookbehind window

Rollup functions read a lookbehind window you can set in square brackets, where i is the current grouping interval:

rate($metric[5i])
max_over_time($metric[1d])

When you omit the window, a rollup reads only the current interval. rate and irate are the exception: they look back max(5 × interval, 5m), so the first buckets of a fine grid still have an earlier sample to rate against.

Instruments

A metric's instrument decides which aggregations it accepts. An aggregation the instrument cannot compute fails the query with <instrument> instrument does not support "<func>".

• Counter measures a value that only grows, such as the number of processed requests. min, max, and avg of a growing total carry no meaning, so the instrument rejects them.
• Gauge measures a value that goes up and down, such as memory utilization. sum exists for Prometheus and AWS compatibility, and it double-counts a gauge that reports more than once per interval.
• PromCounter is a cumulative counter from Prometheus remote write or the Prometheus receiver. It keeps its running total, so wrap it in rate or increase to read a rate.
• Additive measures a value that grows and shrinks and stays meaningful when summed across attributes, such as open connections.
• Summary stores the min, max, sum, and count of observed values. It keeps no buckets, so it has no percentiles.
• Histogram stores buckets, so it answers percentiles and histogram_quantile.

uniq and count work with every instrument, because they count timeseries rather than values.

Bare selector

A selector with no aggregation gets a default one, which depends on the instrument:

Where an aggregation runs

Uptrace pushes an aggregation into ClickHouse when you apply it directly to a selector, which is why sum($metric) scans less data than sum($metric + 0).

count, histogram_count, histogram_quantile, p50…p99, and apdex are pushdown-only: they have no in-process form. Applied anywhere but directly to a metric, they fail with <func>() must be applied to a metric:

# Valid.
p95($srv_duration)

# Invalid — the argument is an expression, not a metric.
p95($a + $b)

Empty buckets

A time bucket with no datapoints reads as absent, and a chart draws a gap. count is the exception: it reads as 0, which makes it suitable for "how many series are reporting" queries.

Aggregate functions

Aggregate functions combine timeseries at each timestamp. Grouping decides which series combine — see Grouping.

quantile keeps the PromQL argument order, with the quantile first. share and zscore come from MetricsQL: they do not reduce a group to one series, so each series keeps its own attributes. A negative value, or a group that sums to 0, gives no share.

# The part of all requests each service handles.
share(sum($requests) by (service_name))

stddev and stdvar measure how far the timeseries spread apart. On a Counter they measure the spread of per-interval deltas, and on a Gauge the spread of the latest readings. The other instruments reject them: a PromCounter keeps a growing total, an Additive holds complementary parts of one whole, and a Summary or a Histogram no longer holds the individual observations.

Counting timeseries

uniq counts distinct values, and it accepts grouping and a value filter:

# Number of timeseries.
uniq($status) as num_checks

# Distinct combinations of two attributes.
uniq($hits, host_name, service_name) as num_timeseries

# Distinct host_name for each service_name.
uniq($hits by (service_name), host_name) as num_timeseries

Rollup functions

Rollup functions read the datapoints of one timeseries inside the lookbehind window. The number of timeseries stays the same.

Use rate or increase to read a PromCounter, which stores a cumulative total:

rate($http_requests_total)
increase($http_requests_total[1h])

A rollup reads grid points, not raw datapoints: the scan first reduces each bucket to one value. So count_over_time counts the buckets that have data, not the datapoints ClickHouse holds.

Rollups with parameters

These rollups also take a number, which must be a literal.

• quantile_over_time keeps the PromQL argument order, with the quantile first. A q outside [0..1] fails the query.
• share_gt_over_time and share_le_over_time add up to 1 at every point with data. The threshold converts into the unit of the metric, as in a comparison, so 100ms works on a metric measured in milliseconds.
• predict_linear accepts a number of seconds or a duration such as 1h, which can be negative. A window with fewer than two points has no slope and reads absent.
• duration_over_time adds up the time between consecutive points that are at most max_gap apart. Without max_gap, only neighbouring grid points join.

# Share of the last 5 minutes when latency stayed above 100ms.
share_gt_over_time($latency[5m], 100ms)

# Free disk space 4 hours from now, extrapolated from the last hour.
predict_linear($fs_free[1h], 4h)

Transform functions

Transform functions run on each datapoint of each timeseries. The number of timeseries and the number of datapoints stay the same.

Rate units

perMin(sum($cache)) and sum($cache) / _minutes give the same result — see Scalars.

Rounding and sign

Exponents and logarithms

Visible range

These transforms read each series over the whole visible range, starting at the first visible point. A point with no data stays absent.

A running function does not carry its last result into a point with no data. Wrap it in keep_last_value to draw a continuous line:

keep_last_value(running_sum(sum($requests)))

Trigonometry

sin, cos, tan, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, and atanh each take one argument and return the named function of every value. deg converts radians to degrees, and rad converts degrees to radians.

Timeseries functions

Timeseries functions rewrite the set of series or fill their gaps. They change how many series a query returns, or which datapoints those series hold.

# The five busiest hosts.
topk(5, sum($requests) by (host_name))

# Fill short gaps instead of drawing them.
keep_last_value(avg($temperature))

With by (...), each group keeps its own n series. The optional third argument of topk_* and bottomk_* adds one series per group that sums the series that were not kept:

# The five services with the most requests, plus one series for the rest.
topk_avg(5, sum($requests) by (service_name), "service_name=other")

Attribute rewriting

These functions rewrite the attributes of each series. They keep the PromQL and MetricsQL label_* names, so a copied query works. Each key and value is a string literal.

Two series that end up with the same attributes fail the query, as in PromQL. Aggregate first:

label_keep(sum($requests) by (service_name), "service_name")

Operators

Arithmetic and comparison

Uptrace supports +, -, *, /, %, ^, the comparisons ==, !=, <, <=, >, >=, and the set operators and, unless, or, and union.

Precedence runs from highest to lowest:

• ^
• *, /, %
• +, -
• ==, !=, <=, <, >=, >
• and, unless
• or, union
• if, ifnot, default

Operators on one level are left-associative, so 2 * 3 % 2 is the same as (2 * 3) % 2. The exception is ^, which is right-associative: 2 ^ 3 ^ 2 is 2 ^ (3 ^ 2).

or keeps a right-side series only where the left side has none with the same attributes. union concatenates both sides without matching attributes at all.

Conditional operators

Calculate a hit rate only when the sample is large enough, and fill the rest with zero:

(sum($misses) / (sum($hits) + sum($misses))
  if (sum($hits) + sum($misses) >= 100)) default 0

Grouping and joining

Group at the function level, at the expression level, or for the whole query:

# Function level.
sum($metric) by (host_name, service_name)
avg(sum($metric) by (cpu, mode)) by (cpu)

# Expression level.
sum($metric1) by (type) / sum($metric2) group by host_name

# Query level, affecting every expression.
$metric1 | $metric2 | group by host_name

Math between series joins them by their matching attributes, and one-to-many and many-to-one joins work without extra syntax:

# One-to-one.
$mem_free + $mem_cached group by host_name

# One-to-many.
$cpu_secs by (mode) / $cpu_secs by (service_name, mode)

When the attribute names differ, rename one side:

$metric1 by (hostname as host) + $metric2 by (host_name as host)

Attribute functions

These functions run on attribute values and are valid only in a grouping expression:

group by lower(service_name) as service
group by extract(host_name, `^uptrace-prod-(\w+)$`) as host
group by replace(host_name, 'uptrace-prod-', '') as host

Expression aliases

Name an expression with as, then reference the name in a later expression of the same query:

metrics:
  - service_cache_redis as $redis
query:
  - $redis{type="hits"} as hits
  - $redis{type="misses"} as misses
  - hits / (hits + misses) as hit_rate

An alias that starts with an underscore takes part in the calculation but does not appear in the result, which keeps helper expressions off the chart:

sum($cache{type="hits"}) as _hits
sum($cache{type="misses"}) as _misses
_misses / (_hits + _misses) as miss_rate

Scalars

See also

• Querying metrics
• PromQL compatibility
• Dashboard YAML templates
• OpenTelemetry Metrics