OpenTelemetry Rust distro for Uptrace

This document explains how to configure OpenTelemetry Rust SDK to export spans and metrics to Uptrace using OTLP/gRPC.

Configuration

You can configure Uptrace client using a DSN (Data Source Name) from the project settings page.

You can also use environment variables to configure the client:

Env var	Description
`UPTRACE_DSN`	A data source that is used to connect to uptrace.dev. For example, `https://<token>@uptrace.dev/<project_id>`.
`OTEL_RESOURCE_ATTRIBUTES`	Key-value pairs to be used as resource attributes. For example, `service.name=myservice,service.version=1.0.0`.
`OTEL_SERVICE_NAME=myservice`	Sets the value of the service.name resource attribute. Takes precedence over `OTEL_RESOURCE_ATTRIBUTES`.
`OTEL_PROPAGATORS`	Propagators to be used as a comma separated list. The default is `tracecontext,baggage`.

See OpenTelemetry documentationopen in new window for details.

Already using OTLP exporter?

If you are already using OTLP exporter, you can continue to use it with Uptrace by changing some configuration options.

To maximize performance and efficiency, consider the following recommendations when configuring OpenTelemetry SDK.

Recommendation	Signals	Significance
Use `BatchSpanProcessor` to export multiple spans in a single request.	All	Essential
Enable `gzip` compression to compress the data before sending and reduce the traffic cost.	All	Essential
Prefer `delta` metrics temporality, because such metrics are smaller and Uptrace must convert `cumulative` metrics to `delta` anyway.	Metrics	Recommended
Use AWS X-Ray ID generator for OpenTelemetry.	Traces, Logs	Optional

To configure OpenTelemetry to send data to Uptrace, use the provided endpoint and pass the DSN via uptrace-dsn header:

# Only for OTLP/gRPC
export OTEL_EXPORTER_OTLP_ENDPOINT="otlp.uptrace.dev:4317"

# Only for OTLP/HTTP
export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.uptrace.dev"

# Pass Uptrace DSN in gRPC/HTTP headers.
export OTEL_EXPORTER_OTLP_HEADERS="uptrace-dsn=https://token@api.uptrace.dev/project_id"

# Enable gzip compression.
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip

# Enable exponential histograms.
export OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=BASE2_EXPONENTIAL_BUCKET_HISTOGRAM

# Prefer delta temporality.
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA

When configuring BatchSpanProcessor, use the following settings:

# Maximum allowed time to export data in milliseconds.
export OTEL_BSP_EXPORT_TIMEOUT=5000

# Maximum batch size.
# Using larger batch sizes can be problematic,
# because Uptrace rejects requests larger than 20MB.
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=10000

# Maximum queue size.
# Increase queue size if you have lots of RAM, for example,
# `10000 * number_of_gigabytes`.
export OTEL_BSP_MAX_QUEUE_SIZE=30000

# Max concurrent exports.
# Setting this to the number of available CPUs might be a good idea.
export OTEL_BSP_MAX_CONCURRENT_EXPORTS=2

Exporting traces

Hereopen in new window is how you can export OpenTelemetry traces to Uptrace following the recommendations above:

use std::error::Error;
use std::thread;
use std::time::Duration;

use opentelemetry::sdk::resource::{
    EnvResourceDetector, SdkProvidedResourceDetector, TelemetryResourceDetector,
};
use opentelemetry::sdk::Resource;
use opentelemetry::trace::TraceError;
use opentelemetry::trace::{TraceContextExt, Tracer};
use opentelemetry::{global, sdk, Key, KeyValue};
use opentelemetry_otlp::WithExportConfig;
use tonic::metadata::MetadataMap;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error + Send + Sync + 'static>> {
    let dsn = std::env::var("UPTRACE_DSN").expect("Error: UPTRACE_DSN not found");
    println!("using DSN: {}", dsn);

    let _ = init_tracer(dsn)?;
    let tracer = global::tracer("app_or_crate_name");

    tracer.in_span("root-span", |cx| {
        thread::sleep(Duration::from_millis(5));

        tracer.in_span("GET /posts/:id", |cx| {
            thread::sleep(Duration::from_millis(10));

            let span = cx.span();
            span.set_attribute(Key::new("http.method").string("GET"));
            span.set_attribute(Key::new("http.route").string("/posts/:id"));
            span.set_attribute(Key::new("http.url").string("http://localhost:8080/posts/123"));
            span.set_attribute(Key::new("http.status_code").i64(200));
        });

        tracer.in_span("SELECT", |cx| {
            thread::sleep(Duration::from_millis(20));

            let span = cx.span();
            span.set_attribute(KeyValue::new("db.system", "mysql"));
            span.set_attribute(KeyValue::new(
                "db.statement",
                "SELECT * FROM posts LIMIT 100",
            ));
        });

        let span = cx.span();
        println!(
            "https://app.uptrace.dev/traces/{}",
            span.span_context().trace_id().to_string()
        );
    });

    global::shutdown_tracer_provider();
    Ok(())
}

fn init_tracer(dsn: String) -> Result<sdk::trace::Tracer, TraceError> {
    let resource = Resource::from_detectors(
        Duration::from_secs(0),
        vec![
            Box::new(SdkProvidedResourceDetector),
            Box::new(EnvResourceDetector::new()),
            Box::new(TelemetryResourceDetector),
        ],
    );

    let mut metadata = MetadataMap::with_capacity(1);
    metadata.insert("uptrace-dsn", dsn.parse().unwrap());

    opentelemetry_otlp::new_pipeline()
        .tracing()
        .with_exporter(
            opentelemetry_otlp::new_exporter()
                .tonic()
                .with_endpoint("https://otlp.uptrace.dev:4317")
                .with_timeout(Duration::from_secs(5))
                .with_metadata(metadata),
        )
        .with_batch_config(
            sdk::trace::BatchConfig::default()
                .with_max_queue_size(30000)
                .with_max_export_batch_size(10000)
                .with_scheduled_delay(Duration::from_millis(5000)),
        )
        .with_trace_config(sdk::trace::config().with_resource(resource))
        .install_batch(opentelemetry::runtime::Tokio)
}

Exporting metrics

Hereopen in new window is how you can export OpenTelemetry metrics to Uptrace following the recommendations above:

use std::error::Error;
use std::thread;
use std::time::Duration;

use opentelemetry::sdk::export::metrics::aggregation::delta_temporality_selector;
use opentelemetry::sdk::metrics::controllers::BasicController;
use opentelemetry::sdk::metrics::selectors;
use opentelemetry::sdk::resource::{
    EnvResourceDetector, SdkProvidedResourceDetector, TelemetryResourceDetector,
};
use opentelemetry::sdk::Resource;
use opentelemetry::{global, metrics, Context};
use opentelemetry_otlp::{ExportConfig, Protocol, WithExportConfig};
use tonic::metadata::MetadataMap;

#[tokio::main]
async fn main() -> Result<(), Box<dyn Error + Send + Sync + 'static>> {
    let dsn = std::env::var("UPTRACE_DSN").expect("Error: UPTRACE_DSN not found");
    println!("using DSN: {}", dsn);

    let _ = init_metrics(dsn)?;
    let meter = global::meter("app_or_crate_name");
    let histogram = meter.f64_histogram("ex.com.three").init();

    let cx = Context::new();
    for _i in 1..100000 {
        histogram.record(&cx, 1.3, &[]);
        thread::sleep(Duration::from_millis(100));
    }

    Ok(())
}

fn init_metrics(dsn: String) -> metrics::Result<BasicController> {
    let resource = Resource::from_detectors(
        Duration::from_secs(0),
        vec![
            Box::new(SdkProvidedResourceDetector),
            Box::new(EnvResourceDetector::new()),
            Box::new(TelemetryResourceDetector),
        ],
    );

    let mut metadata = MetadataMap::with_capacity(1);
    metadata.insert("uptrace-dsn", dsn.parse().unwrap());

    let export_config = ExportConfig {
        endpoint: "https://otlp.uptrace.dev:4317".to_string(),
        timeout: Duration::from_secs(10),
        protocol: Protocol::Grpc,
    };
    let exporter = opentelemetry_otlp::new_exporter()
        .tonic()
        .with_export_config(export_config)
        .with_metadata(metadata);

    opentelemetry_otlp::new_pipeline()
        .metrics(
            selectors::simple::inexpensive(),
            delta_temporality_selector(),
            opentelemetry::runtime::Tokio,
        )
        .with_exporter(exporter)
        .with_period(Duration::from_secs(15))
        .with_timeout(Duration::from_secs(5))
        .with_resource(resource)
        .build()
}

Rust tracing

tokio-rs/tracingopen in new window is framework for instrumenting Rust programs to collect structured, event-based diagnostic information. It is used by many popular Rust frameworks and libraries.

You can integrate tokio-rs/tracing with OpenTelemetry using the tracing_opentelemetryopen in new window crate:

use tracing::{error, span};
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::Registry;

// configure OpenTelemetry

// Create a tracing layer with the configured tracer
let tracer = init_tracer(dsn)?;

// Create a tracing layer with the configured tracer
let telemetry = tracing_opentelemetry::layer().with_tracer(tracer);

// Use the tracing subscriber `Registry`, or any other subscriber
// that impls `LookupSpan`
let subscriber = Registry::default().with(telemetry);

// Trace executed code
tracing::subscriber::with_default(subscriber, || {
    // Spans will be sent to the configured OpenTelemetry exporter
    let root = span!(tracing::Level::TRACE, "app_start", work_units = 2);
    let _enter = root.enter();

    error!("This event will be logged in the root span.");
});

Troubleshooting

You can set OpenTelemetry error handler like this:

opentelemetry::global::set_error_handler(|error| {
    ::tracing::error!(
        target: "opentelemetry",
        "OpenTelemetry error occurred: {:#}",
        anyhow::anyhow!(error),
    );
})
.expect("to be able to set error handler");

If you are getting OpenTelemetry trace error occurred. cannot send span to the batch span processor because the channel is full, you can tweak BatchSpanProcessor config using env variables:

export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=10000
export OTEL_BSP_MAX_QUEUE_SIZE=30000
export OTEL_BSP_MAX_CONCURRENT_EXPORTS=2

# OpenTelemetry Rust distro for Uptrace

# Configuration

# Already using OTLP exporter?

# Exporting traces

# Exporting metrics

# Rust tracing

# Troubleshooting