OpenTelemetry sampling: head-based and tail-based

OpenTelemetry Sampling reduces the cost and verbosity of tracing by reducing the number of created (sampled) spans. In terms of performance, sampling can save CPU cycles and memory required to collect, process, and export spans.

• What is sampling?
• Sampling: when and where
• Sampling probability
• Head-based sampling
  • Head-based sampling in OpenTelemetry
  • OpenTelemetry samplers
• Rate-limiting sampling
• Tail-based sampling
• Probability-based sampling
• Uptrace

What is sampling?

Sampling is used in distributed tracing to control the volume of data collected and sent to the tracing backend. It helps to balance the tradeoff between data volume and trace accuracy.

In distributed tracing, a request generates spans as it flows through a system. Spans represent individual operations or events that occur during the processing of that request. These spans can become quite numerous in a complex system, and sending them all to the tracing backend can result in significant overhead and storage costs.

Sampling involves making decisions about which spans to record and which to discard.

Sampling: when and where

Sampling may happen in different stages of spans processing:

• When a trace is created - head-based sampling;
• When a trace is received by a backend - rate-limiting sampling;
• When a complete trace is available - tail-based sampling.

The choice of sampling strategy depends on several factors, including the desired level of observability, available resources, and the specific use case of the system.

Sampling probability

Sampling provides a sampling probability which enables accurate statistical counting of all spans using only a portion of sampled spans. For example, if the sampling probability is 50% and the number of sampled spans is 10, then the adjusted (total) number of spans is 10 / 50% = 20.

Name	Side	Adjusted count	Accuracy
Head-based sampling	Client-side	Yes	100%
Rate-limiting sampling	Server-side	Yes	<90%
Tail-based sampling	Server-side	Yes	<90%

Head-based sampling

Head-based sampling makes the sampling decision as early as possible and propagates it to other participants using the context. This allows saving CPU and memory resources by not collecting any telemetry data for dropped spans (operations).

Head-based sampling is the simplest, most accurate, and most reliable sampling method which you should prefer over all other methods.

A disadvantage of head-based sampling is that you can't sample spans with errors, because that information is not available when spans are created. To address that concern, you can use tail-based sampling.

Head-based sampling also does not account for traffic spikes and may collect more data than desired. This is where rate-limiting sampling becomes handy.

Head-based sampling in OpenTelemetry

OpenTelemetry has 2 span properties responsible for client sampling:

• IsRecording - when false, span discards attributes, events, links etc.
• Sampled - when false, OpenTelemetry drops the span.

You should check IsRecording property to avoid collecting expensive telemetry data.

if span.IsRecording() {
    // collect expensive data
}

Sampler is a function that accepts a root span about to be created. The function returns a sampling decision which must be one of:

• Drop - trace is dropped. IsRecording = false, Sampled = false.
• RecordOnly - trace is recorded but not sampled. IsRecording = true, Sampled = false.
• RecordAndSample - trace is recorded and sampled. IsRecording = true, Sampled = true.

By default, OpenTelemetry samples all traces, but you can configure it to sample a portion of traces. In that case, backends use the sampling probability to adjust the number of spans.

OpenTelemetry samplers

AlwaysOn sampler samples every trace, for example, a new trace will be started and exported for every request.

AlwaysOff sampler samples no traces or, in other words, drops all traces. You can use this sampler to perform load testing or to temporarily disable tracing.

TraceIDRatioBased sampler uses a trace id to sample a fraction of traces, for example, 20% of traces.

Parent-based sampler is a composite sampler which behaves differently based on the parent of the span. When you start a new trace, sampler makes a decision whether or not to sample it and propagates the decision down to other services.

Rate-limiting sampling

Rate-limiting sampling happens on the server side and ensures that you don't exceed certain limits, for example, it allows to sample 10 or less traces per seconds.

Rate-limiting sampling supports adjusted counts but the accuracy is rather low. To achieve better results and improve performance, you should use rate-limiting sampling together with head-based sampling which is more efficient and accurate.

Most backends (including Uptrace) automatically apply rate-limiting sampling when necessary.

Tail-based sampling

With head-based sampling the sampling decision is made upfront and usually at random. Head-based sampling can't sample failed or unusually long operations, because that information is only available at the end of a trace.

With tail-based sampling we delay the sampling decision until all spans of a trace are available which enables better sampling decisions based on all data from the trace. For example, we can sample failed or unusually long traces.

Most OpenTelemetry backends automatically apply tail-based sampling when necessary, but you can also use OpenTelemetry Collector with tailsamplingprocessor to configure sampling according to your needs.

Probability-based sampling

Probability-based sampling randomly selects a subset of traces to record based on a configured probability or sampling rate. For example, you can set a sampling rate of 10%, which means that only 10% of the traces are recorded and the rest are discarded.

Probability-based sampling is useful when you want to reduce the amount of trace data while still maintaining a representative sample of system behavior. It helps strike a balance between overhead and the level of observability you need.

Here is how you can configure a probability-based sampler in OpenTelemetry Go:

import "go.opentelemetry.io/contrib/samplers/probability/consistent"

sampler := consistent.ParentProbabilityBased(
	consistent.ProbabilityBased(0.5), // sample 50% of traces
)

uptrace.ConfigureOpentelemetry(
	uptrace.WithTraceSampler(sampler),

	// Other options
)

Uptrace

Uptrace is an open source APM with an intuitive query builder, rich dashboards, alerting rules, and integrations for most languages and frameworks. It can process billions of spans and metrics on a single server and allows to monitor your applications at 10x lower cost.

Uptrace uses ClickHouse database to store traces, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.

You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.

• Jaeger vs OpenTelemetry
• OpenTelemetry vs Prometheus