OpenTelemetry Go distro for Uptrace

This document explains how to configure OpenTelemetry Go SDK to export spans and metrics to Uptrace using OTLP/gRPC.

To learn about OpenTelemetry API, see OpenTelemetry Go Tracing APIopen in new window and OpenTelemetry Go Metrics APIopen in new window.

Uptrace Go

uptrace-goopen in new window is a thin wrapper over opentelemetry-goopen in new window that configures OpenTelemetry SDK to export data to Uptrace. It does not add any new functionality and is provided only for your convenience.

To install uptrace-go:

go get github.com/uptrace/uptrace-go

Configuration

You can configure Uptrace client using a DSN (Data Source Name) from the project settings page.

import "github.com/uptrace/uptrace-go/uptrace"

uptrace.ConfigureOpentelemetry(
    // copy your project DSN here or use UPTRACE_DSN env var
    //uptrace.WithDSN("https://FIXME@api.uptrace.dev?grpc=4317"),

	uptrace.WithServiceName("myservice"),
	uptrace.WithServiceVersion("v1.0.0"),
	uptrace.WithDeploymentEnvironment("production"),
)

You can find the full list of available options at pkg.go.devopen in new window.

OptionDescription
WithDSNA data source that specifies Uptrace project credentials. For example, https://<token>@api.uptrace.dev?grpc=4317.
WithServiceNameservice.name resource attribute. For example, myservice.
WithServiceVersionservice.version resource attribute. For example, 1.0.0.
WithDeploymentEnvironmentdeployment.environment resource attribute. For example, production.
WithResourceAttributesAny other resource attributes.
WithResourceDetectorsConfigures additional resource detectors, for example, EC2 detector or GCE detector.
WithResourceResource contains attributes representing an entity that produces telemetry. Resource attributes are copied to all spans and events.

You can also use environment variables to configure the client:

Env varDescription
UPTRACE_DSNA data source that is used to connect to uptrace.dev. For example, https://<token>@uptrace.dev/<project_id>.
OTEL_RESOURCE_ATTRIBUTESKey-value pairs to be used as resource attributes. For example, service.name=myservice,service.version=1.0.0.
OTEL_SERVICE_NAME=myserviceSets the value of the service.name resource attribute. Takes precedence over OTEL_RESOURCE_ATTRIBUTES.
OTEL_PROPAGATORSPropagators to be used as a comma separated list. The default is tracecontext,baggage.

See OpenTelemetry documentationopen in new window for details.

Quickstart

Spend 5 minutes to install OpenTelemetry distro, generate your first trace, and click the link in your terminal to view the trace.

go get github.com/uptrace/uptrace-go
package main

import (
	"context"
	"errors"
	"fmt"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/trace"

	"github.com/uptrace/uptrace-go/uptrace"
)

func main() {
	ctx := context.Background()

	// Configure OpenTelemetry with sensible defaults.
	uptrace.ConfigureOpentelemetry(
		// copy your project DSN here or use UPTRACE_DSN env var
		// uptrace.WithDSN("https://FIXME@api.uptrace.dev?grpc=4317"),

		uptrace.WithServiceName("myservice"),
		uptrace.WithServiceVersion("1.0.0"),
	)
	// Send buffered spans and free resources.
	defer uptrace.Shutdown(ctx)

	// Create a tracer. Usually, tracer is a global variable.
	tracer := otel.Tracer("app_or_package_name")

	// Create a root span (a trace) to measure some operation.
	ctx, main := tracer.Start(ctx, "main-operation")
	// End the span when the operation we are measuring is done.
	defer main.End()

	// The passed ctx carries the parent span (main).
	// That is how OpenTelemetry manages span relations.
	_, child1 := tracer.Start(ctx, "GET /posts/:id")
	child1.SetAttributes(
		attribute.String("http.method", "GET"),
		attribute.String("http.route", "/posts/:id"),
		attribute.String("http.url", "http://localhost:8080/posts/123"),
		attribute.Int("http.status_code", 200),
	)
	if err := errors.New("dummy error"); err != nil {
		child1.RecordError(err, trace.WithStackTrace(true))
		child1.SetStatus(codes.Error, err.Error())
		child1.End()
	}

	_, child2 := tracer.Start(ctx, "SELECT")
	child2.SetAttributes(
		attribute.String("db.system", "mysql"),
		attribute.String("db.statement", "SELECT * FROM posts LIMIT 100"),
	)
	child2.End()

	fmt.Printf("trace: %s\n", uptrace.TraceURL(main))
}
  • Step 3. Run the code to get a link for the generated trace:
go run main.go
trace: https://app.uptrace.dev/traces/<trace_id>
  • Step 4. Follow the link to view the trace:

Basic trace

Already using OTLP exporter?

If you are already using OTLP exporter, you can continue to use it with Uptrace by changing some configuration options.

To maximize performance and efficiency, consider the following recommendations when configuring OpenTelemetry SDK.

RecommendationSignalsSignificance
Use BatchSpanProcessor to export multiple spans in a single request.AllEssential
Enable gzip compression to compress the data before sending and reduce the traffic cost.AllEssential
Prefer delta metrics temporality, because such metrics are smaller and Uptrace must convert cumulative metrics to delta anyway.MetricsRecommended
Prefer Protobuf encoding over JSON.AllRecommended
Use AWS X-Ray ID generator for OpenTelemetry.Traces, LogsOptional

To configure OpenTelemetry to send data to Uptrace, use the provided endpoint and pass the DSN via uptrace-dsn header:

TransportEndpointPort
gRPChttps://otlp.uptrace.dev:43174317
HTTPShttps://otlp.uptrace.dev443

Most languages allow to configure OTLP exporter using environment variables:

# Uncomment the appropriate protocol for your programming language.
# Only for OTLP/gRPC
#export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.uptrace.dev:4317"
# Only for OTLP/HTTP
#export OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp.uptrace.dev"

# Pass Uptrace DSN in gRPC/HTTP headers.
export OTEL_EXPORTER_OTLP_HEADERS="uptrace-dsn=https://FIXME@api.uptrace.dev?grpc=4317"

# Enable gzip compression.
export OTEL_EXPORTER_OTLP_COMPRESSION=gzip

# Enable exponential histograms.
export OTEL_EXPORTER_OTLP_METRICS_DEFAULT_HISTOGRAM_AGGREGATION=BASE2_EXPONENTIAL_BUCKET_HISTOGRAM

# Prefer delta temporality.
export OTEL_EXPORTER_OTLP_METRICS_TEMPORALITY_PREFERENCE=DELTA

When configuring BatchSpanProcessor, use the following settings:

# Maximum allowed time to export data in milliseconds.
export OTEL_BSP_EXPORT_TIMEOUT=10000

# Maximum batch size.
# Using larger batch sizes can be problematic,
# because Uptrace rejects requests larger than 20MB.
export OTEL_BSP_MAX_EXPORT_BATCH_SIZE=10000

# Maximum queue size.
# Increase queue size if you have lots of RAM, for example,
# `10000 * number_of_gigabytes`.
export OTEL_BSP_MAX_QUEUE_SIZE=30000

# Max concurrent exports.
# Setting this to the number of available CPUs might be a good idea.
export OTEL_BSP_MAX_CONCURRENT_EXPORTS=2

Exporting traces

Hereopen in new window is how you can export traces to Uptrace following the recommendations above:

package main

import (
	"context"
	"fmt"
	"os"

	"go.opentelemetry.io/contrib/propagators/aws/xray"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
)

func main() {
	ctx := context.Background()

	dsn := os.Getenv("UPTRACE_DSN")
	if dsn == "" {
		panic("UPTRACE_DSN environment variable is required")
	}
	fmt.Println("using DSN:", dsn)

	exporter, err := otlptracehttp.New(
		ctx,
		otlptracehttp.WithEndpoint("otlp.uptrace.dev"),
		otlptracehttp.WithHeaders(map[string]string{
			// Set the Uptrace DSN here or use UPTRACE_DSN env var.
			"uptrace-dsn": dsn,
		}),
		otlptracehttp.WithCompression(otlptracehttp.GzipCompression),
	)
	if err != nil {
		panic(err)
	}

	bsp := sdktrace.NewBatchSpanProcessor(exporter,
		sdktrace.WithMaxQueueSize(10_000),
		sdktrace.WithMaxExportBatchSize(10_000))
	// Call shutdown to flush the buffers when program exits.
	defer bsp.Shutdown(ctx)

	resource, err := resource.New(ctx,
		resource.WithFromEnv(),
		resource.WithTelemetrySDK(),
		resource.WithHost(),
		resource.WithAttributes(
			attribute.String("service.name", "myservice"),
			attribute.String("service.version", "1.0.0"),
		))
	if err != nil {
		panic(err)
	}

	tracerProvider := sdktrace.NewTracerProvider(
		sdktrace.WithResource(resource),
		sdktrace.WithIDGenerator(xray.NewIDGenerator()),
	)
	tracerProvider.RegisterSpanProcessor(bsp)

	// Install our tracer provider and we are done.
	otel.SetTracerProvider(tracerProvider)

	tracer := otel.Tracer("app_or_package_name")
	ctx, span := tracer.Start(ctx, "main")
	defer span.End()

	fmt.Printf("trace: https://app.uptrace.dev/traces/%s\n", span.SpanContext().TraceID())
}

Exporting metrics

Hereopen in new window is how you can export metrics to Uptrace following the recommendations above:

package main

import (
	"context"
	"fmt"
	"os"
	"time"

	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc"
	"go.opentelemetry.io/otel/metric"
	sdkmetric "go.opentelemetry.io/otel/sdk/metric"
	"go.opentelemetry.io/otel/sdk/metric/metricdata"
	"go.opentelemetry.io/otel/sdk/resource"
	"google.golang.org/grpc/encoding/gzip"
)

func main() {
	ctx := context.Background()

	dsn := os.Getenv("UPTRACE_DSN")
	if dsn == "" {
		panic("UPTRACE_DSN environment variable is required")
	}
	fmt.Println("using DSN:", dsn)

	exporter, err := otlpmetricgrpc.New(ctx,
		otlpmetricgrpc.WithEndpoint("otlp.uptrace.dev:4317"),
		otlpmetricgrpc.WithHeaders(map[string]string{
			// Set the Uptrace DSN here or use UPTRACE_DSN env var.
			"uptrace-dsn": dsn,
		}),
		otlpmetricgrpc.WithCompressor(gzip.Name),
		otlpmetricgrpc.WithTemporalitySelector(preferDeltaTemporalitySelector),
	)
	if err != nil {
		panic(err)
	}

	reader := sdkmetric.NewPeriodicReader(
		exporter,
		sdkmetric.WithInterval(15*time.Second),
	)

	resource, err := resource.New(ctx,
		resource.WithFromEnv(),
		resource.WithTelemetrySDK(),
		resource.WithHost(),
		resource.WithAttributes(
			attribute.String("service.name", "myservice"),
			attribute.String("service.version", "1.0.0"),
		))
	if err != nil {
		panic(err)
	}

	provider := sdkmetric.NewMeterProvider(
		sdkmetric.WithReader(reader),
		sdkmetric.WithResource(resource),
	)
	otel.SetMeterProvider(provider)

	meter := provider.Meter("app_or_package_name")
	counter, _ := meter.Int64Counter(
		"uptrace.demo.counter_name",
		metric.WithUnit("1"),
		metric.WithDescription("counter description"),
	)

	fmt.Println("exporting data to Uptrace...")
	for {
		counter.Add(ctx, 1)
		time.Sleep(time.Millisecond)
	}
}

func preferDeltaTemporalitySelector(kind sdkmetric.InstrumentKind) metricdata.Temporality {
	switch kind {
	case sdkmetric.InstrumentKindCounter,
		sdkmetric.InstrumentKindObservableCounter,
		sdkmetric.InstrumentKindHistogram:
		return metricdata.DeltaTemporality
	default:
		return metricdata.CumulativeTemporality
	}
}

Exporting logs

Hereopen in new window is how you can export logs to Uptrace following the recommendations above:

package main

import (
	"context"
	"fmt"
	"log/slog"
	"os"
	"time"

	"go.opentelemetry.io/contrib/bridges/otelslog"
	"go.opentelemetry.io/contrib/propagators/aws/xray"
	"go.opentelemetry.io/otel"
	"go.opentelemetry.io/otel/attribute"
	"go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploghttp"
	"go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracehttp"
	"go.opentelemetry.io/otel/log/global"
	sdklog "go.opentelemetry.io/otel/sdk/log"
	"go.opentelemetry.io/otel/sdk/resource"
	sdktrace "go.opentelemetry.io/otel/sdk/trace"
	"go.opentelemetry.io/otel/trace"
)

func main() {
	ctx := context.Background()

	dsn := os.Getenv("UPTRACE_DSN")
	if dsn == "" {
		panic("UPTRACE_DSN environment variable is required")
	}
	fmt.Println("using DSN:", dsn)

	resource, err := resource.New(ctx,
		resource.WithFromEnv(),
		resource.WithTelemetrySDK(),
		resource.WithHost(),
		resource.WithAttributes(
			attribute.String("service.name", "myservice"),
			attribute.String("service.version", "1.0.0"),
		))
	if err != nil {
		panic(err)
	}

	shutdownTracing := configureTracing(ctx, dsn, resource)
	defer shutdownTracing()

	shutdownLogging := configureLogging(ctx, dsn, resource)
	defer shutdownLogging()

	tracer := otel.Tracer("app_or_package_name")
	logger := otelslog.NewLogger("app_or_package_name")

	ctx, main := tracer.Start(ctx, "main-operation", trace.WithSpanKind(trace.SpanKindServer))
	defer main.End()

	logger.ErrorContext(ctx, "hello world", slog.String("error", "error message"))

	fmt.Printf("trace: https://app.uptrace.dev/traces/%s\n", main.SpanContext().TraceID())
}

func configureLogging(ctx context.Context, dsn string, resource *resource.Resource) func() {
	exp, err := otlploghttp.New(ctx,
		otlploghttp.WithEndpoint("otlp.uptrace.dev"),
		otlploghttp.WithHeaders(map[string]string{
			"uptrace-dsn": dsn,
		}),
		otlploghttp.WithCompression(otlploghttp.GzipCompression),
	)
	if err != nil {
		panic(err)
	}

	bsp := sdklog.NewBatchProcessor(exp,
		sdklog.WithMaxQueueSize(10_000),
		sdklog.WithExportMaxBatchSize(10_000),
		sdklog.WithExportInterval(10*time.Second),
		sdklog.WithExportTimeout(10*time.Second),
	)

	provider := sdklog.NewLoggerProvider(
		sdklog.WithProcessor(bsp),
		sdklog.WithResource(resource),
	)

	global.SetLoggerProvider(provider)

	return func() {
		provider.Shutdown(ctx)
	}
}

func configureTracing(ctx context.Context, dsn string, resource *resource.Resource) func() {
	exporter, err := otlptracehttp.New(
		ctx,
		otlptracehttp.WithEndpoint("otlp.uptrace.dev"),
		otlptracehttp.WithHeaders(map[string]string{
			"uptrace-dsn": dsn,
		}),
		otlptracehttp.WithCompression(otlptracehttp.GzipCompression),
	)
	if err != nil {
		panic(err)
	}

	bsp := sdktrace.NewBatchSpanProcessor(exporter,
		sdktrace.WithMaxQueueSize(10_000),
		sdktrace.WithMaxExportBatchSize(10_000),
	)

	tracerProvider := sdktrace.NewTracerProvider(
		sdktrace.WithResource(resource),
		sdktrace.WithIDGenerator(xray.NewIDGenerator()),
	)
	tracerProvider.RegisterSpanProcessor(bsp)

	otel.SetTracerProvider(tracerProvider)

	return func() {
		tracerProvider.Shutdown(ctx)
	}
}

Serverless

AWS Lambda

See OpenTelemetry Go Lambda.

Vercel

On Vercelopen in new window, you need to configure OpenTelemetry in the init function and ForceFlush spans when the Vercel handler exits.

package handler

import (
	"fmt"
	"net/http"

    "go.opentelemetry.io/otel"
	"github.com/uptrace/uptrace-go/uptrace"
)

var tracer = otel.Tracer("app_or_package_name")

func init() {
	uptrace.ConfigureOpentelemetry(...)
}

func Handler(w http.ResponseWriter, req *http.Request) {
	ctx := req.Context()

	// Flush buffered spans.
	defer uptrace.ForceFlush(ctx)

	ctx, span := tracer.Start(ctx, "handler-name")
	defer span.End()

	fmt.Fprintf(w, "<h1>Hello from Go!</h1>")
}

Sampling

You can reduce the number of created (sampled) spans by configuring head-based samplingopen in new window:

import "go.opentelemetry.io/contrib/samplers/probability/consistent"

sampler := consistent.ParentProbabilityBased(
	consistent.ProbabilityBased(0.5), // sample 50% of traces
)

uptrace.ConfigureOpentelemetry(
	uptrace.WithTraceSampler(sampler),

	// Other options
)

By default, uptrace-go samples all spans.

Resource detectors

By default, uptrace-go uses host and environment resource detectors, but you can configure it to use additional detectors, for example:

import (
	"github.com/uptrace/uptrace-go/uptrace"
	"go.opentelemetry.io/contrib/detectors/aws/ec2"
)

uptrace.ConfigureOpentelemetry(
	// copy your project DSN here or use UPTRACE_DSN env var
	//uptrace.WithDSN("https://FIXME@api.uptrace.dev?grpc=4317"),

	uptrace.WithServiceName("myservice"),
	uptrace.WithServiceVersion("1.0.0"),

	uptrace.WithResourceDetectors(ec2.NewResourceDetector()),
)
AWS

See AWS detectorsopen in new window.

EC2

import "go.opentelemetry.io/contrib/detectors/aws/ec2"

ec2ResourceDetector := ec2.NewResourceDetector()

ECS

import "go.opentelemetry.io/contrib/detectors/aws/ecs"

ecsResourceDetector := ecs.NewResourceDetector()

EKS

import "go.opentelemetry.io/contrib/detectors/aws/eks"

eksResourceDetector := eks.NewResourceDetector()
Google Cloud

See GCP detectorsopen in new window.

Cloud Run

import "go.opentelemetry.io/contrib/detectors/gcp"

cloudRunResourceDetector := gcp.NewCloudRun()

GCE

import "go.opentelemetry.io/contrib/detectors/gcp"

gceResourceDetector := gcp.GCE{}

GKE

import "go.opentelemetry.io/contrib/detectors/gcp"

gkeResourceDetector := gcp.GKE{}

Errors monitoring

To monitor errors with a stack trace, use the RecordError API:

import (
	"go.opentelemetry.io/otel/codes"
	"go.opentelemetry.io/otel/trace"
)

ctx, span := tracer.Start(ctx, "operation-name")
defer span.End()

span.RecordError(errors.New("oh my"), trace.WithStackTrace(true))
span.SetStatus(codes.Error, err.Error())

See OpenTelemetry Go Tracing APIopen in new window for details.

Logs monitoring

To monitor logs, use OpenTelemetry instrumentations for popular logging libraries:

If that is not possible, see Monitoring Logs for more options such as Vector or FluentBit.

SDK logging

By default, OpenTelemetry Go SDK logs error messages to stderr. You can discard or redirect those logs by providing an error handler:

import "go.opentelemetry.io/otel"

otel.SetErrorHandler(otel.ErrorHandlerFunc(func(err error) {
    // ignore the error
}))

Baggage

You can use package baggagecopyopen in new window to read key/values stored in baggageopen in new window in the starting span's parent context and add them as attributes to the span.

import "go.opentelemetry.io/contrib/processors/baggagecopy"

uptrace.ConfigureOpentelemetry(
	WithSpanProcessor(
		baggagecopy.NewSpanProcessor(nil),
    ),
)

What's next?

Next, instrument more operations to get a more detailed picture. Try to prioritize network calls, disk operations, database queries, error and logs.

You can also create your own instrumentations using OpenTelemetry Go Tracing APIopen in new window.

Last Updated: