Metrics query language reference

This page lists every function, operator, and selector the Uptrace metrics query language accepts. To learn how to write queries, start with Querying metrics; to adapt existing Prometheus queries, see PromQL compatibility.

Selectors

A query references a metric through an alias that starts with $:

metrics:
  - system_filesystem_usage as $fs_usage
query:
  - sum($fs_usage)

Attribute filters

Filter a selector by attributes inside curly braces:

You can also filter every expression in a query at once with where, which supports =, !=, <, <=, >, >=, ~, !~, like, not like, in, and not in:

$hits | $misses | where host_name = 'localhost'

Value filter

The _value pseudo-attribute filters timeseries by the datapoint value, not by an attribute. Use it to count series in a given state:

uniq($status{_value=1}) as num_up | uniq($status{_value=0}) as num_down

Time qualifier

$metric._time reads the datapoint timestamp instead of the value. It accepts only min and max, which return the first and the last time the metric reported:

min($cache._time) | max($cache._time)

Offset

offset shifts the query window. A negative offset looks ahead of the evaluation time:

$http_requests_total offset 5m
$http_requests_total offset -5m

Lookbehind window

Rollup functions read a lookbehind window you can set in square brackets, where i is the current grouping interval:

rate($metric[5i])
max_over_time($metric[1d])

When you omit the window, a rollup reads only the current interval. rate and irate are the exception: they look back max(5 × interval, 5m), so the first buckets of a fine grid still have an earlier sample to rate against.

Instruments

A metric's instrument decides which aggregations it accepts. An aggregation the instrument cannot compute fails the query with <instrument> instrument does not support "<func>".

• Counter measures a value that only grows, such as the number of processed requests. min, max, and avg of a growing total carry no meaning, so the instrument rejects them.
• Gauge measures a value that goes up and down, such as memory utilization. sum exists for Prometheus and AWS compatibility, and it double-counts a gauge that reports more than once per interval.
• PromCounter is a cumulative counter from Prometheus remote write or the Prometheus receiver. It keeps its running total, so wrap it in rate or increase to read a rate.
• Additive measures a value that grows and shrinks and stays meaningful when summed across attributes, such as open connections.
• Summary stores the min, max, sum, and count of observed values. It keeps no buckets, so it has no percentiles.
• Histogram stores buckets, so it answers percentiles and histogram_quantile.

uniq and count work with every instrument, because they count timeseries rather than values.

Bare selector

A selector with no aggregation gets a default one, which depends on the instrument:

Where an aggregation runs

Uptrace pushes an aggregation into ClickHouse when you apply it directly to a selector, which is why sum($metric) scans less data than sum($metric + 0).

count, histogram_count, histogram_quantile, p50…p99, and apdex are pushdown-only: they have no in-process form. Applied anywhere but directly to a metric, they fail with <func>() must be applied to a metric:

# Valid.
p95($srv_duration)

# Invalid — the argument is an expression, not a metric.
p95($a + $b)

Empty buckets

A time bucket with no datapoints reads as absent, and a chart draws a gap. count is the exception: it reads as 0, which makes it suitable for "how many series are reporting" queries.

Aggregate functions

Aggregate functions combine timeseries at each timestamp. Grouping decides which series combine — see Grouping.

stddev and stdvar measure how far the timeseries spread apart. On a Counter they measure the spread of per-interval deltas, and on a Gauge the spread of the latest readings. The other instruments reject them: a PromCounter keeps a growing total, an Additive holds complementary parts of one whole, and a Summary or a Histogram no longer holds the individual observations.

Counting timeseries

uniq counts distinct values, and it accepts grouping and a value filter:

# Number of timeseries.
uniq($status) as num_checks

# Distinct combinations of two attributes.
uniq($hits, host_name, service_name) as num_timeseries

# Distinct host_name for each service_name.
uniq($hits by (service_name), host_name) as num_timeseries

Rollup functions

Rollup functions read the datapoints of one timeseries inside the lookbehind window. The number of timeseries stays the same.

Use rate or increase to read a PromCounter, which stores a cumulative total:

rate($http_requests_total)
increase($http_requests_total[1h])

Transform functions

Transform functions run on each datapoint of each timeseries. The number of timeseries and the number of datapoints stay the same.

Rate units

perMin(sum($cache)) and sum($cache) / _minutes give the same result — see Scalars.

Rounding and sign

Exponents and logarithms

Trigonometry

sin, cos, tan, asin, acos, atan, sinh, cosh, tanh, asinh, acosh, and atanh each take one argument and return the named function of every value. deg converts radians to degrees, and rad converts degrees to radians.

Timeseries functions

Timeseries functions rewrite the set of series or fill their gaps. They change how many series a query returns, or which datapoints those series hold.

# The five busiest hosts.
topk(sum($requests by (host_name)), 5)

# Fill short gaps instead of drawing them.
keep_last_value(avg($temperature))

Operators

Arithmetic and comparison

Uptrace supports +, -, *, /, %, ^, the comparisons ==, !=, <, <=, >, >=, and the set operators and, unless, or, and union.

Precedence runs from highest to lowest:

• ^
• *, /, %
• +, -
• ==, !=, <=, <, >=, >
• and, unless
• or, union
• if, ifnot, default

Operators on one level are left-associative, so 2 * 3 % 2 is the same as (2 * 3) % 2. The exception is ^, which is right-associative: 2 ^ 3 ^ 2 is 2 ^ (3 ^ 2).

or keeps a right-side series only where the left side has none with the same attributes. union concatenates both sides without matching attributes at all.

Conditional operators

Calculate a hit rate only when the sample is large enough, and fill the rest with zero:

(sum($misses) / (sum($hits) + sum($misses))
  if (sum($hits) + sum($misses) >= 100)) default 0

Grouping and joining

Group at the function level, at the expression level, or for the whole query:

# Function level.
sum($metric) by (host_name, service_name)
avg(sum($metric) by (cpu, mode)) by (cpu)

# Expression level.
sum($metric1) by (type) / sum($metric2) group by host_name

# Query level, affecting every expression.
$metric1 | $metric2 | group by host_name

Math between series joins them by their matching attributes, and one-to-many and many-to-one joins work without extra syntax:

# One-to-one.
$mem_free + $mem_cached group by host_name

# One-to-many.
$cpu_secs by (mode) / $cpu_secs by (service_name, mode)

When the attribute names differ, rename one side:

$metric1 by (hostname as host) + $metric2 by (host_name as host)

Attribute functions

These functions run on attribute values and are valid only in a grouping expression:

group by lower(service_name) as service
group by extract(host_name, `^uptrace-prod-(\w+)$`) as host
group by replace(host_name, 'uptrace-prod-', '') as host

Expression aliases

Name an expression with as, then reference the name in a later expression of the same query:

metrics:
  - service_cache_redis as $redis
query:
  - $redis{type="hits"} as hits
  - $redis{type="misses"} as misses
  - hits / (hits + misses) as hit_rate

An alias that starts with an underscore takes part in the calculation but does not appear in the result, which keeps helper expressions off the chart:

sum($cache{type="hits"}) as _hits
sum($cache{type="misses"}) as _misses
_misses / (_hits + _misses) as miss_rate

Scalars

See also

• Querying metrics
• PromQL compatibility
• Dashboard YAML templates
• OpenTelemetry Metrics