OpenTelemetry Go-Zero monitoring [otelzero]

Go-Zero has built-in OpenTelemetry support, making it easy to add distributed tracing to your microservices without additional instrumentation libraries. By enabling telemetry in your configuration, you can gain insights into application performance, troubleshoot issues, and analyze behavior across your entire service mesh.

Quick Setup

Minimal configuration:

Telemetry:
  Name: my-service
  Endpoint: localhost:14317
  Sampler: 1.0
  Batcher: otlpgrpc
  OtlpHeaders:
    uptrace-dsn: '<FIXME>'

Go-Zero automatically instruments HTTP handlers and RPC calls when telemetry is configured—no additional middleware or code changes required.

What is Go-Zero?

Go-Zero is an open-source microservices framework for the Go programming language. It is designed to simplify the development of high-performance, scalable, and reliable microservices with built-in support for:

• API gateway and RPC services
• Service discovery and load balancing
• Rate limiting and circuit breaking
• Distributed tracing via OpenTelemetry

Go-Zero's integrated approach to observability means you don't need external instrumentation libraries—telemetry is configured directly in your service configuration.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize and simplify the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry supports multiple programming languages and platforms, making it suitable for a wide range of applications and environments.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization. Configuration management is simplified through OpenTelemetry environment variables, allowing teams to adjust telemetry settings across development, staging, and production environments without modifying application code.

The OpenTelemetry architecture provides a modular, vendor-neutral approach: Go-Zero's built-in support uses the OpenTelemetry Go SDK under the hood, so traces from Go-Zero services are fully compatible with any OpenTelemetry-based backend.

Go-Zero Telemetry Configuration

Go-Zero uses a Telemetry configuration block that maps to its internal trace.Config struct:

type Config struct {
    Name         string            `json:",optional"`
    Endpoint     string            `json:",optional"`
    Sampler      float64           `json:",default=1.0"`
    Batcher      string            `json:",default=otlpgrpc,options=zipkin|otlpgrpc|otlphttp|file"`
    OtlpHeaders  map[string]string `json:",optional"`
    OtlpHttpPath string            `json:",optional"`
    OtlpHttpSecure bool            `json:",optional"`
    Disabled     bool              `json:",optional"`
}

Usage

You can instrument your Go-Zero application by enabling OpenTelemetry in your YAML configuration. No code changes are required—Go-Zero automatically instruments HTTP handlers and RPC calls.

API Service Configuration

To start monitoring your Go-Zero API service, add the Telemetry block to your config. If you don't have an Uptrace DSN, follow the Get started guide.

Name: user-api
Host: 0.0.0.0
Port: 8080
Mode: dev

Telemetry:
  Name: user-api
  Endpoint: localhost:14317
  Sampler: 1.0
  Batcher: otlpgrpc
  OtlpHeaders:
    uptrace-dsn: '<FIXME>'

RPC Service Configuration

For RPC services, the configuration is similar:

Name: user-rpc
ListenOn: 0.0.0.0:9000
Mode: dev

Telemetry:
  Name: user-rpc
  Endpoint: localhost:14317
  Sampler: 1.0
  Batcher: otlpgrpc
  OtlpHeaders:
    uptrace-dsn: '<FIXME>'

Loading Configuration

Go-Zero automatically initializes telemetry when loading your configuration:

package main

import (
    "github.com/zeromicro/go-zero/core/conf"
    "github.com/zeromicro/go-zero/rest"
)

func main() {
    var c rest.RestConf
    conf.MustLoad("etc/config.yaml", &c)

    // Telemetry is automatically initialized
    server := rest.MustNewServer(c)
    defer server.Stop()

    // Register your handlers
    server.Start()
}

You can also find a complete Docker example on GitHub.

Creating Custom Spans

While Go-Zero automatically instruments HTTP and RPC handlers, you can add custom spans to trace specific operations like database queries or external API calls:

import (
    "context"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
)

var tracer = otel.Tracer("my-service")

func GetUser(ctx context.Context, userID string) (*User, error) {
    ctx, span := tracer.Start(ctx, "GetUser")
    defer span.End()

    span.SetAttributes(attribute.String("user.id", userID))

    user, err := queryUserFromDB(ctx, userID)
    if err != nil {
        span.SetAttributes(attribute.Bool("user.found", false))
        return nil, err
    }

    span.SetAttributes(attribute.Bool("user.found", true))
    return user, nil
}

Pass the context.Context from Go-Zero handlers to your functions so that custom spans appear as children of the automatic HTTP/RPC spans.

Recording Errors

Record errors on spans to mark them as failed and capture error details:

import (
    "context"

    "go.opentelemetry.io/otel/codes"
    "go.opentelemetry.io/otel/trace"
)

func ProcessOrder(ctx context.Context, order *Order) error {
    span := trace.SpanFromContext(ctx)

    if err := validateOrder(order); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, err.Error())
        return err
    }

    if err := saveOrder(ctx, order); err != nil {
        span.RecordError(err)
        span.SetStatus(codes.Error, "failed to save order")
        return err
    }

    return nil
}

OTLP HTTP Exporter

If your backend only accepts OTLP over HTTP (instead of gRPC), switch the Batcher and adjust the endpoint:

Telemetry:
  Name: my-service
  Endpoint: api.uptrace.dev:443
  Sampler: 1.0
  Batcher: otlphttp
  OtlpHttpPath: /v1/traces
  OtlpHttpSecure: true
  OtlpHeaders:
    uptrace-dsn: '<FIXME>'

Go-Zero passes Endpoint directly to otlptracehttp.WithEndpoint(), which expects a scheme-less host:port value. Use OtlpHttpSecure: true to enable HTTPS instead of including https:// in the endpoint.

FAQ

Does Go-Zero support metrics and logs via OpenTelemetry? Go-Zero's built-in telemetry focuses on distributed tracing. For metrics, use the OpenTelemetry Go SDK directly or export Go-Zero's built-in Prometheus metrics. For logs, integrate a structured logger like slog or Zap.

How do I control sampling in production? Set Sampler to a value between 0.0 and 1.0. For high-traffic services, start with 0.1 (10%) and adjust based on your tracing volume and budget. A value of 1.0 traces every request.

Can I trace calls between Go-Zero services? Yes. Go-Zero automatically propagates trace context across HTTP and gRPC boundaries. When service A calls service B, the trace context is injected into outgoing requests and extracted by the receiving service, creating a connected distributed trace.

What happens if the telemetry endpoint is unavailable? Go-Zero's OTLP exporter buffers spans and retries delivery. If the endpoint remains unavailable, spans are eventually dropped. Your application continues to run normally—telemetry failures do not affect request handling.

How do I disable tracing in development? Set Disabled: true in the Telemetry configuration block, or remove the Telemetry section entirely.

What is Uptrace?

Uptrace is an open source APM for OpenTelemetry that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules, notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

What's next?

With OpenTelemetry tracing enabled in your Go-Zero application, you can visualize traces, analyze performance, and identify bottlenecks in your microservices architecture.

Next steps to enhance your observability:

• Create custom spans using the OpenTelemetry Go Tracing API
• Add database instrumentation with GORM or database/sql
• Explore other Go frameworks like Gin or gRPC
• Set up the OpenTelemetry Collector for production deployments