OpenTelemetry Sampling [.NET]

What is sampling?

Sampling is a process that restricts the amount of traces that are generated by a system. In high-volume applications, collecting 100% of traces can be expensive and unnecessary. Sampling allows you to collect a representative subset of traces while reducing costs and performance overhead.

.NET sampling implementation

OpenTelemetry .NET SDK provides built-in sampling capabilities through the Sampler class. The sampling decision is made at the beginning of a trace (head-based sampling) and affects the entire trace tree.

Built-in samplers

AlwaysOnSampler

Samples 100% of traces. Useful for development and debugging:

using OpenTelemetry;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new AlwaysOnSampler())
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

AlwaysOffSampler

Disables tracing entirely. Useful for temporarily turning off tracing:

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new AlwaysOffSampler())
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

TraceIdRatioBasedSampler

Samples a percentage of traces based on the trace ID. The sampling rate is consistent across all services:

// Sample 10% of traces
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new TraceIdRatioBasedSampler(0.1)) // 10% sampling rate
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

ParentBasedSampler

Respects the sampling decision of the parent span. If there's no parent, it uses a root sampler:

// Parent-based sampler with 25% root sampling
var rootSampler = new TraceIdRatioBasedSampler(0.25);
var parentBasedSampler = new ParentBasedSampler(rootSampler);

builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(parentBasedSampler)
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

Configuration in .NET

Environment variable configuration

You can configure sampling using environment variables:

bash

# Set sampling ratio to 10%
OTEL_TRACES_SAMPLER=traceidratio
OTEL_TRACES_SAMPLER_ARG=0.1

# Always sample
OTEL_TRACES_SAMPLER=always_on

# Never sample
OTEL_TRACES_SAMPLER=always_off

# Parent-based sampling
OTEL_TRACES_SAMPLER=parentbased_traceidratio
OTEL_TRACES_SAMPLER_ARG=0.2

Programmatic configuration

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("my-api", "1.0.0"))
    .WithTracing(tracing => tracing
        .SetSampler(new TraceIdRatioBasedSampler(0.1)) // 10% sampling
        .AddSource("MyApp")
        .AddAspNetCoreInstrumentation(options =>
        {
            options.RecordException = true;
        })
        .AddHttpClientInstrumentation()
        .AddOtlpExporter()
    );

Custom sampler implementation

You can create custom sampling logic by implementing the Sampler class:

using OpenTelemetry.Trace;
using System.Diagnostics;

public class CustomSampler : Sampler
{
    private readonly double _samplingRatio;
    private readonly Random _random = new Random();

    public CustomSampler(double samplingRatio)
    {
        _samplingRatio = samplingRatio;
    }

    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        // Always sample error traces
        var tags = samplingParameters.Tags;
        if (tags != null)
        {
            foreach (var tag in tags)
            {
                if (tag.Key == "error" && tag.Value?.ToString() == "true")
                {
                    return SamplingResult.Create(SamplingDecision.RecordAndSample);
                }
            }
        }

        // Sample high-priority operations
        var operationName = samplingParameters.Name;
        if (operationName.Contains("critical") || operationName.Contains("payment"))
        {
            return SamplingResult.Create(SamplingDecision.RecordAndSample);
        }

        // Use probability sampling for other traces
        if (_random.NextDouble() < _samplingRatio)
        {
            return SamplingResult.Create(SamplingDecision.RecordAndSample);
        }

        return SamplingResult.Create(SamplingDecision.Drop);
    }
}

// Register custom sampler
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new CustomSampler(0.1))
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

Advanced .NET sampling scenarios

Sampling based on request attributes

public class AttributeBasedSampler : Sampler
{
    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        var tags = samplingParameters.Tags;
        if (tags != null)
        {
            foreach (var tag in tags)
            {
                // Always sample requests to admin endpoints
                if (tag.Key == "http.route" && tag.Value?.ToString()?.Contains("/admin") == true)
                {
                    return SamplingResult.Create(SamplingDecision.RecordAndSample);
                }

                // Sample 50% of API requests
                if (tag.Key == "http.route" && tag.Value?.ToString()?.StartsWith("/api") == true)
                {
                    var traceId = samplingParameters.TraceId;
                    var hash = traceId.GetHashCode();
                    return Math.Abs(hash) % 100 < 50
                        ? SamplingResult.Create(SamplingDecision.RecordAndSample)
                        : SamplingResult.Create(SamplingDecision.Drop);
                }
            }
        }

        // Default: don't sample
        return SamplingResult.Create(SamplingDecision.Drop);
    }
}

Time-based sampling

public class TimeBasedSampler : Sampler
{
    private readonly Dictionary<DateTime, int> _samplesPerMinute = new();
    private readonly int _maxSamplesPerMinute;
    private readonly object _lock = new object();

    public TimeBasedSampler(int maxSamplesPerMinute)
    {
        _maxSamplesPerMinute = maxSamplesPerMinute;
    }

    public override SamplingResult ShouldSample(in SamplingParameters samplingParameters)
    {
        lock (_lock)
        {
            var currentMinute = new DateTime(DateTime.UtcNow.Year, DateTime.UtcNow.Month,
                DateTime.UtcNow.Day, DateTime.UtcNow.Hour, DateTime.UtcNow.Minute, 0);

            if (!_samplesPerMinute.ContainsKey(currentMinute))
            {
                _samplesPerMinute[currentMinute] = 0;

                // Clean up old entries
                var keysToRemove = _samplesPerMinute.Keys
                    .Where(k => k < currentMinute.AddMinutes(-5))
                    .ToList();
                foreach (var key in keysToRemove)
                {
                    _samplesPerMinute.Remove(key);
                }
            }

            if (_samplesPerMinute[currentMinute] < _maxSamplesPerMinute)
            {
                _samplesPerMinute[currentMinute]++;
                return SamplingResult.Create(SamplingDecision.RecordAndSample);
            }

            return SamplingResult.Create(SamplingDecision.Drop);
        }
    }
}

// Usage: limit to 100 samples per minute
builder.Services.AddOpenTelemetry()
    .WithTracing(tracing => tracing
        .SetSampler(new TimeBasedSampler(100))
        .AddAspNetCoreInstrumentation()
        .AddOtlpExporter()
    );

.NET production deployment

ASP.NET Core application

using OpenTelemetry;
using OpenTelemetry.Resources;
using OpenTelemetry.Trace;

var builder = WebApplication.CreateBuilder(args);

// Configure services
builder.Services.AddControllers();

// Configure OpenTelemetry with sampling
builder.Services.AddOpenTelemetry()
    .ConfigureResource(resource => resource
        .AddService("my-api", "1.0.0")
        .AddAttributes(new Dictionary<string, object>
        {
            ["deployment.environment"] = builder.Environment.EnvironmentName,
            ["service.instance.id"] = Environment.MachineName
        }))
    .WithTracing(tracing => tracing
        .SetSampler(GetSampler(builder.Environment))
        .AddSource("MyApi")
        .AddAspNetCoreInstrumentation(options =>
        {
            options.RecordException = true;
            options.Filter = httpContext =>
            {
                // Don't trace health checks
                return !httpContext.Request.Path.StartsWithSegments("/health");
            };
        })
        .AddHttpClientInstrumentation()
        .AddEntityFrameworkCoreInstrumentation()
        .AddOtlpExporter()
    );

var app = builder.Build();

static Sampler GetSampler(IWebHostEnvironment environment)
{
    return environment.IsDevelopment()
        ? new AlwaysOnSampler()  // Sample everything in dev
        : new TraceIdRatioBasedSampler(0.1);  // 10% in production
}

app.MapControllers();
app.Run();

Controller with manual span creation

[ApiController]
[Route("api/[controller]")]
public class OrdersController : ControllerBase
{
    private static readonly ActivitySource ActivitySource = new("MyApi");
    private readonly IOrderService _orderService;

    public OrdersController(IOrderService orderService)
    {
        _orderService = orderService;
    }

    [HttpPost]
    public async Task<IActionResult> CreateOrder([FromBody] CreateOrderRequest request)
    {
        using var activity = ActivitySource.StartActivity("CreateOrder");
        activity?.SetTag("order.customer_id", request.CustomerId);
        activity?.SetTag("order.items_count", request.Items.Count);

        try
        {
            var order = await _orderService.CreateOrderAsync(request);
            activity?.SetTag("order.id", order.Id);
            activity?.SetStatus(ActivityStatusCode.Ok);

            return Ok(order);
        }
        catch (Exception ex)
        {
            activity?.SetStatus(ActivityStatusCode.Error, ex.Message);
            activity?.RecordException(ex);
            throw;
        }
    }
}

Monitoring sampling in .NET

Check sampling ratio

public class SamplingMetricsMiddleware
{
    private readonly RequestDelegate _next;
    private readonly ILogger<SamplingMetricsMiddleware> _logger;
    private static long _totalRequests = 0;
    private static long _sampledRequests = 0;

    public SamplingMetricsMiddleware(RequestDelegate next, ILogger<SamplingMetricsMiddleware> logger)
    {
        _next = next;
        _logger = logger;
    }

    public async Task InvokeAsync(HttpContext context)
    {
        Interlocked.Increment(ref _totalRequests);

        var activity = Activity.Current;
        if (activity != null && activity.IsAllDataRequested)
        {
            Interlocked.Increment(ref _sampledRequests);
        }

        // Log sampling ratio every 1000 requests
        if (_totalRequests % 1000 == 0)
        {
            var ratio = (double)_sampledRequests / _totalRequests;
            _logger.LogInformation("Sampling ratio: {Ratio:P} ({Sampled}/{Total})",
                ratio, _sampledRequests, _totalRequests);
        }

        await _next(context);
    }
}

What's next?

Context PropagationConfigure context propagation for .NET apps using W3C TraceContext, baggage, and custom propagators to keep traces connected end to end.

Zero-codeAutomatically instrument .NET applications without code changes using OpenTelemetry .NET auto-instrumentation. Export traces, metrics, and logs to Uptrace via OTLP.