OpenTelemetry Golang gRPC monitoring [otelgrpc]

OpenTelemetry gRPC is an OpenTelemetry instrumentation for collecting telemetry data from gRPC applications.

Quick Reference

What's collected:

• Traces: Full RPC call traces with method, service, status code
• Metrics: Request duration, message size, call counts
• Context propagation: Automatic trace context across services

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize and simplify the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry supports multiple programming languages and platforms, making it suitable for a wide range of applications and environments.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization. The framework supports configuration through OpenTelemetry environment variables, enabling easy customization of exporters, sampling rates, and resource attributes across different environments.

What is gRPC?

gRPC is a cross-platform open source high performance Remote Procedure Call (RPC) framework for Golang. gRPC is ideal for lightweight microservices where efficiency is critical.

gRPC provides efficient communication between distributed systems, allowing them to interact and exchange data in a reliable and scalable manner.

gRPC instrumentation

OpenTelemetry gRPC is easy to use and provides a simple API for instrumenting gRPC applications, making it possible for developers to quickly add observability to their gRPC applications without having to write a lot of code.

To install otelgrpc instrumentation:

go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

• Example
• Reference

Usage

Instrumenting gRPC Client

import (
    "crypto/tls"
    "google.golang.org/grpc"
    "google.golang.org/grpc/credentials"
    "google.golang.org/grpc/credentials/insecure"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

// For insecure connections (development)
conn, err := grpc.NewClient(target,
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

// For TLS connections (production)
conn, err := grpc.NewClient(target,
    grpc.WithTransportCredentials(credentials.NewTLS(&tls.Config{})),
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

Note: grpc.WithInsecure() is deprecated. Use grpc.WithTransportCredentials(insecure.NewCredentials()) for insecure connections.

Instrumenting gRPC Server

import (
    "google.golang.org/grpc"
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

server := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

Metadata and baggage

On the client side, you can use gRPC metadata as a baggage:

import "google.golang.org/grpc/metadata"

md := metadata.Pairs(
    "key1", "value1",
    "key2", "value2",
)
ctx := metadata.NewOutgoingContext(context.Background(), md)

On the server side, you can extract gRPC metadata from the incoming request:

import "google.golang.org/grpc/metadata"

md, ok := metadata.FromIncomingContext(ctx); ok {
    fmt.Println(md)
}

Or use OpenTelemetry baggage API:

import "go.opentelemetry.io/otel/baggage"

baggage := baggage.FromContext(ctx)

Configuration Options

Custom Handler Options

You can configure the otelgrpc handlers with various options:

import (
    "go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

// Configure client handler with custom options
clientHandler := otelgrpc.NewClientHandler(
    otelgrpc.WithTracerProvider(tracerProvider),
    otelgrpc.WithMeterProvider(meterProvider),
    otelgrpc.WithMessageEvents(otelgrpc.ReceivedEvents), // Record message events
)

conn, err := grpc.NewClient(target,
    grpc.WithTransportCredentials(insecure.NewCredentials()),
    grpc.WithStatsHandler(clientHandler),
)

Available options:

• WithTracerProvider() - Use custom tracer provider
• WithMeterProvider() - Use custom meter provider
• WithMessageEvents() - Record sent/received message events in spans
• WithMetricAttributes() - Add custom attributes to metrics

Server Handler Configuration

// Configure server handler with custom options
serverHandler := otelgrpc.NewServerHandler(
    otelgrpc.WithTracerProvider(tracerProvider),
    otelgrpc.WithMeterProvider(meterProvider),
)

server := grpc.NewServer(
    grpc.StatsHandler(serverHandler),
)

Collected Metrics

The otelgrpc instrumentation automatically collects the following metrics:

Client Metrics

• rpc.client.duration - Duration of RPC calls (histogram)
  • Attributes: rpc.method, rpc.service, rpc.grpc.status_code
• rpc.client.request.size - Size of request messages (histogram)
• rpc.client.response.size - Size of response messages (histogram)
• rpc.client.requests_per_rpc - Number of requests per RPC (histogram, streaming only)
• rpc.client.responses_per_rpc - Number of responses per RPC (histogram, streaming only)

Server Metrics

• rpc.server.duration - Duration of RPC calls (histogram)
  • Attributes: rpc.method, rpc.service, rpc.grpc.status_code
• rpc.server.request.size - Size of request messages (histogram)
• rpc.server.response.size - Size of response messages (histogram)
• rpc.server.requests_per_rpc - Number of requests per RPC (histogram, streaming only)
• rpc.server.responses_per_rpc - Number of responses per RPC (histogram, streaming only)

All metrics follow OpenTelemetry semantic conventions for RPC.

Troubleshooting

Deprecated grpc.WithInsecure() Error

Problem: Code fails with "grpc.WithInsecure is deprecated" or similar error.

Solution: Replace with grpc.WithTransportCredentials(insecure.NewCredentials()):

// ❌ Old (deprecated)
conn, err := grpc.Dial(target, grpc.WithInsecure())

// ✅ New (correct)
import "google.golang.org/grpc/credentials/insecure"

conn, err := grpc.NewClient(target,
    grpc.WithTransportCredentials(insecure.NewCredentials()),
)

Using Deprecated Interceptors

Problem: Old code uses UnaryClientInterceptor() or UnaryServerInterceptor().

Solution: Migrate to StatsHandler-based instrumentation:

// ❌ Old (deprecated interceptors)
conn, err := grpc.Dial(target,
    grpc.WithUnaryInterceptor(otelgrpc.UnaryClientInterceptor()),
    grpc.WithStreamInterceptor(otelgrpc.StreamClientInterceptor()),
)

// ✅ New (StatsHandler)
conn, err := grpc.NewClient(target,
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

StatsHandler provides better performance and more accurate metrics compared to interceptors.

Missing Trace Context

Problem: Trace context not propagating between client and server.

Solution: Ensure both client and server use otelgrpc instrumentation:

// Client side - automatically injects trace context
conn, err := grpc.NewClient(target,
    grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

// Server side - automatically extracts trace context
server := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

The instrumentation uses gRPC metadata to propagate trace context automatically. No manual context injection needed.

What is Uptrace?

Uptrace is a OpenTelemetry backend that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues. For APM capabilities, see the OpenTelemetry Go guide and compare with top APM tools.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

What's next?

Your gRPC services now have comprehensive observability with OpenTelemetry tracing. Monitor RPC calls, track latencies, and debug cross-service communication issues. For web frameworks, check out Gin or Echo instrumentation.