OpenTelemetry Sampling [.NET]

What is sampling?

Sampling is a process that restricts the amount of traces that are generated by a system. In high-volume applications, collecting 100% of traces can be expensive and unnecessary. Sampling allows you to collect a representative subset of traces while reducing costs and performance overhead.

.NET sampling implementation

OpenTelemetry .NET SDK provides built-in sampling capabilities through the Sampler class. The sampling decision is made at the beginning of a trace (head-based sampling) and affects the entire trace tree.

Built-in samplers

AlwaysOnSampler

Samples 100% of traces. Useful for development and debugging:

using OpenTelemetry;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new AlwaysOnSampler())
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

AlwaysOffSampler

Disables tracing entirely. Useful for temporarily turning off tracing:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new AlwaysOffSampler())
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

TraceIdRatioBasedSampler

Samples a percentage of traces based on the trace ID. The sampling rate is consistent across all services:

// Sample 10% of traces
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new TraceIdRatioBasedSampler(0.1)) // 10% sampling rate
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

ParentBasedSampler

Respects the sampling decision of the parent span. If there's no parent, it uses a root sampler:

// Parent-based sampler with 25% root sampling
var rootSampler = new TraceIdRatioBasedSampler(0.25);
var parentBasedSampler = new ParentBasedSampler(rootSampler);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(parentBasedSampler)
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

Configuration in .NET

Environment variable configuration

You can configure sampling using environment variables:

bash

# Set sampling ratio to 10%
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

# Always sample
OTEL_TRACES_SAMPLER=always_on

# Never sample
OTEL_TRACES_SAMPLER=always_off

# Parent-based sampling
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.2

Programmatic configuration

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("my-api", "1.0.0"))
    .WithTracing(tracing => tracing
        .SetSampler(new TraceIdRatioBasedSampler(0.1)) // 10% sampling
        .AddSource("MyApp")
        .AddAspNetCoreInstrumentation(options =>
        {
            options.RecordException = true;
        })
        .AddHttpClientInstrumentation()
        .AddOtlpExporter()
    );

Custom sampler implementation

You can create custom sampling logic by implementing the Sampler class:

using OpenTelemetry.Trace;
using System.Diagnostics;

public class CustomSampler : Sampler
{
    private readonly double _samplingRatio;
    private readonly Random _random = new Random();

    public CustomSampler(double samplingRatio)
    {
        _samplingRatio = samplingRatio;
    }

    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        // Always sample error traces
        var tags = samplingParameters.Tags;
        if (tags != null)
        {
            foreach (var tag in tags)
            {
                if (tag.Key == "error" && tag.Value?.ToString() == "true")
                {
                    return SamplingResult.Create(SamplingDecision.RecordAndSample);
                }
            }
        }

        // Sample high-priority operations
        var operationName = samplingParameters.Name;
        if (operationName.Contains("critical") || operationName.Contains("payment"))
        {
            return SamplingResult.Create(SamplingDecision.RecordAndSample);
        }

        // Use probability sampling for other traces
        if (_random.NextDouble() < _samplingRatio)
        {
            return SamplingResult.Create(SamplingDecision.RecordAndSample);
        }

        return SamplingResult.Create(SamplingDecision.Drop);
    }
}

// Register custom sampler
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new CustomSampler(0.1))
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

Advanced .NET sampling scenarios

Sampling based on request attributes

public class AttributeBasedSampler : Sampler
{
    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        var tags = samplingParameters.Tags;
        if (tags != null)
        {
            foreach (var tag in tags)
            {
                // Always sample requests to admin endpoints
                if (tag.Key == "http.route" && tag.Value?.ToString()?.Contains("/admin") == true)
                {
                    return SamplingResult.Create(SamplingDecision.RecordAndSample);
                }

                // Sample 50% of API requests
                if (tag.Key == "http.route" && tag.Value?.ToString()?.StartsWith("/api") == true)
                {
                    var traceId = samplingParameters.TraceId;
                    var hash = traceId.GetHashCode();
                    return Math.Abs(hash) % 100 < 50
                        ? SamplingResult.Create(SamplingDecision.RecordAndSample)
                        : SamplingResult.Create(SamplingDecision.Drop);
                }
            }
        }

        // Default: don't sample
        return SamplingResult.Create(SamplingDecision.Drop);
    }
}

Time-based sampling

public class TimeBasedSampler : Sampler
{
    private readonly Dictionary<DateTime, int> _samplesPerMinute = new();
    private readonly int _maxSamplesPerMinute;
    private readonly object _lock = new object();

    public TimeBasedSampler(int maxSamplesPerMinute)
    {
        _maxSamplesPerMinute = maxSamplesPerMinute;
    }

    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        lock (_lock)
        {
            var currentMinute = new DateTime(DateTime.UtcNow.Year, DateTime.UtcNow.Month,
                DateTime.UtcNow.Day, DateTime.UtcNow.Hour, DateTime.UtcNow.Minute, 0);

            if (!_samplesPerMinute.ContainsKey(currentMinute))
            {
                _samplesPerMinute[currentMinute] = 0;

                // Clean up old entries
                var keysToRemove = _samplesPerMinute.Keys
                    .Where(k => k < currentMinute.AddMinutes(-5))
                    .ToList();
                foreach (var key in keysToRemove)
                {
                    _samplesPerMinute.Remove(key);
                }
            }

            if (_samplesPerMinute[currentMinute] < _maxSamplesPerMinute)
            {
                _samplesPerMinute[currentMinute]++;
                return SamplingResult.Create(SamplingDecision.RecordAndSample);
            }

            return SamplingResult.Create(SamplingDecision.Drop);
        }
    }
}

// Usage: limit to 100 samples per minute
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new TimeBasedSampler(100))
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

.NET production deployment

ASP.NET Core application

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

// Configure services
builder.Services.AddControllers();

// Configure OpenTelemetry with sampling
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("my-api", "1.0.0")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment"] = builder.Environment.EnvironmentName,
            ["service.instance.id"] = Environment.MachineName
        }))
    .WithTracing(tracing => tracing
        .SetSampler(GetSampler(builder.Environment))
        .AddSource("MyApi")
        .AddAspNetCoreInstrumentation(options =>
        {
            options.RecordException = true;
            options.Filter = httpContext =>
            {
                // Don't trace health checks
                return !httpContext.Request.Path.StartsWithSegments("/health");
            };
        })
        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddOtlpExporter()
    );

var app = builder.Build();

static Sampler GetSampler(IWebHostEnvironment environment)
{
    return environment.IsDevelopment()
        ? new AlwaysOnSampler()  // Sample everything in dev
        : new TraceIdRatioBasedSampler(0.1);  // 10% in production
}

app.MapControllers();
app.Run();

Controller with manual span creation

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    private static readonly ActivitySource ActivitySource = new("MyApi");
    private readonly IOrderService _orderService;

    public OrdersController(IOrderService orderService)
    {
        _orderService = orderService;
    }

    [HttpPost]
    public async Task<IActionResult> CreateOrder([FromBody] CreateOrderRequest request)
    {
        using var activity = ActivitySource.StartActivity("CreateOrder");
        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.items_count", request.Items.Count);

        try
        {
            var order = await _orderService.CreateOrderAsync(request);
            activity?.SetTag("order.id", order.Id);
            activity?.SetStatus(ActivityStatusCode.Ok);

            return Ok(order);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

Monitoring sampling in .NET

Check sampling ratio

public class SamplingMetricsMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<SamplingMetricsMiddleware> _logger;
    private static long _totalRequests = 0;
    private static long _sampledRequests = 0;

    public SamplingMetricsMiddleware(RequestDelegate next, ILogger<SamplingMetricsMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        Interlocked.Increment(ref _totalRequests);

        var activity = Activity.Current;
        if (activity != null && activity.IsAllDataRequested)
        {
            Interlocked.Increment(ref _sampledRequests);
        }

        // Log sampling ratio every 1000 requests
        if (_totalRequests % 1000 == 0)
        {
            var ratio = (double)_sampledRequests / _totalRequests;
            _logger.LogInformation("Sampling ratio: {Ratio:P} ({Sampled}/{Total})",
                ratio, _sampledRequests, _totalRequests);
        }

        await _next(context);
    }
}

OpenTelemetry APM

Uptrace is a OpenTelemetry backend that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace Overview

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

What's next?

Context PropagationConfigure context propagation for .NET apps using W3C TraceContext, baggage, and custom propagators to keep traces connected end to end.

Get startedLearn how to monitor your Erlang/Elixir applications with OpenTelemetry and Uptrace.