OpenTelemetry Go Sampling

TL;DR for production

Use consistent probability sampling from the contrib package for accurate span-to-metrics and complete traces across services
Start with 10% sampling and adjust based on traffic volume and observability needs
Wrap with ParentBased to respect parent sampling decisions in distributed systems

import "go.opentelemetry.io/contrib/samplers/probability/consistent"

sampler := consistent.ParentProbabilityBased(
    consistent.ProbabilityBased(0.1), // 10% sampling
)

// With uptrace-go
uptrace.ConfigureOpentelemetry(
    uptrace.WithTraceSampler(sampler),
)

// Or with standard SDK
provider := trace.NewTracerProvider(
    trace.WithSampler(sampler),
)

Sampling is a process that restricts the amount of spans that are generated by a system. In high-volume applications, collecting 100% of traces can be expensive and unnecessary. Sampling allows you to collect a representative subset of traces while reducing costs and performance overhead.

OpenTelemetry Go SDK provides head-based sampling capabilities where the sampling decision is made at the beginning of a trace. By default, the tracer provider uses a ParentBased sampler with the AlwaysSample sampler.

Configuring sampling with uptrace-go

If you're using the uptrace-go distribution, configure sampling with the WithTraceSampler option. By default, uptrace-go samples all spans.

import (
    "go.opentelemetry.io/contrib/samplers/probability/consistent"
    "github.com/uptrace/uptrace-go/uptrace"
)

sampler := consistent.ParentProbabilityBased(
    consistent.ProbabilityBased(0.5), // sample 50% of traces
)

uptrace.ConfigureOpentelemetry(
    uptrace.WithTraceSampler(sampler),

    uptrace.WithServiceName("myservice"),
    uptrace.WithServiceVersion("1.0.0"),
)

Built-in samplers

The Go SDK provides these built-in samplers for common use cases:

AlwaysSample

Samples every trace. Useful for development environments but use with caution in production:

import "go.opentelemetry.io/otel/sdk/trace"

provider := trace.NewTracerProvider(
    trace.WithSampler(trace.AlwaysSample()),
)

NeverSample

Samples no traces. Useful for completely disabling tracing:

provider := trace.NewTracerProvider(
    trace.WithSampler(trace.NeverSample()),
)

TraceIDRatioBased

Samples a fraction of traces based on the trace ID. The fraction should be between 0.0 and 1.0:

// Sample 10% of traces
provider := trace.NewTracerProvider(
    trace.WithSampler(trace.TraceIDRatioBased(0.1)),
)

ParentBased

A sampler decorator that behaves differently based on the parent of the span. If the span has no parent, the decorated sampler is used to make the sampling decision. If the span has a parent, the sampler follows the parent's sampling decision:

// ParentBased with TraceIDRatioBased root sampler
provider := trace.NewTracerProvider(
    trace.WithSampler(trace.ParentBased(trace.TraceIDRatioBased(0.1))),
)

Environment variables

For built-in samplers, you can use environment variables:

bash

# Built-in TraceIDRatioBased sampler with 50% sampling
export OTEL_TRACES_SAMPLER="traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.5"

# Built-in ParentBased with TraceIDRatioBased
export OTEL_TRACES_SAMPLER="parentbased_traceidratio"
export OTEL_TRACES_SAMPLER_ARG="0.1"

# Always sample
export OTEL_TRACES_SAMPLER="always_on"

# Never sample
export OTEL_TRACES_SAMPLER="always_off"

Consistent probability sampler (recommended)

For production use, we recommend using the consistent probability based sampler from the official contrib package. This sampler implements the OpenTelemetry specification for consistent probability sampling and records sampling information in the tracestate.

Why use consistent probability sampling?

The consistent probability sampler propagates two values via the tracestate: "p-value" (sampling probability) and "r-value" (randomness source). This approach provides several benefits:

Consistent decisions: All services make the same sampling decision for a trace
Accurate metrics: Spans can be accurately counted using span-to-metrics pipelines
Complete traces: Traces tend to be complete even when spans make independent sampling decisions
Cross-service compatibility: Works correctly across different services and vendors

Basic usage

import (
    "go.opentelemetry.io/contrib/samplers/probability/consistent"
    "go.opentelemetry.io/otel/sdk/trace"
)

// Recommended: Use consistent probability sampling
sampler := consistent.ProbabilityBased(0.1) // Sample 10% of traces

// Wrap with ParentBased for proper parent respect
provider := trace.NewTracerProvider(
    trace.WithSampler(consistent.ParentProbabilityBased(sampler)),
)

Parent-based consistent sampling

import "go.opentelemetry.io/contrib/samplers/probability/consistent"

// Parent-based consistent sampler with different child behaviors
sampler := consistent.ParentProbabilityBased(
    consistent.ProbabilityBased(0.1), // Root sampler
    trace.WithRemoteParentSampled(consistent.ProbabilityBased(0.05)),
    trace.WithLocalParentSampled(consistent.ProbabilityBased(0.15)),
)

Note: Environment variable configuration does not support the consistent probability sampler. Use programmatic configuration.

Performance optimization

Use span.IsRecording() to skip expensive operations on non-sampled spans:

func handleRequest(ctx context.Context) {
    tracer := otel.Tracer("my-service")
    ctx, span := tracer.Start(ctx, "handle-request")
    defer span.End()

    // Only add expensive attributes if the span is recording
    if span.IsRecording() {
        span.SetAttributes(
            attribute.String("user.agent", getUserAgent()),
            attribute.String("request.body", serializeBody()),
        )
    }
}

Best practices

Use consistent probability sampling: Enables accurate span-to-metrics pipelines and complete traces
Start conservative: Begin with 1-5% sampling and increase based on needs
Monitor sampling effectiveness: Track rates to ensure meaningful data capture
Use environment-based configuration: Different rates for dev, staging, and production