Monitoring Golang gRPC with OpenTelemetry

OpenTelemetry gRPC is an OpenTelemetry instrumentation for collecting telemetry data from gRPC applications.

What is OpenTelemetry?

OpenTelemetryopen in new window is an open source and vendor-neutral API for OpenTelemetry tracingopen in new window, logsopen in new window, and metricsopen in new window.

OpenTelemetry provides a set of APIs, libraries, and instrumentation tools to instrument applications and collect data about their behavior and performance.

The goal of OpenTelemetry is to provide a standardized and vendor agnostic way to instrument applications for observability, for example, many open source tracing toolsopen in new window already support OpenTelemetry.

What is gRPC?

gRPC is a cross-platform open source high performance Remote Procedure Call (RPC) framework for Golang. gRPC is ideal for lightweight microservices where efficiency is critical.

gRPC provides efficient communication between distributed systems, allowing them to interact and exchange data in a reliable and scalable manner.

gRPC instrumentation

OpenTelemetry gRPC is easy to use and provides a simple API for instrumenting gRPC applications, making it possible for developers to quickly add observability to their gRPC applications without having to write a lot of code.

To install otelgrpc instrumentation:

go get go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc

Usage

To instrument gRPC client:

import (
	"google.golang.org/grpc"
	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

conn, err := grpc.Dial(target,
	grpc.WithInsecure(),
	grpc.WithStatsHandler(otelgrpc.NewClientHandler()),
)

To instrument gRPC server:

import (
	"google.golang.org/grpc"
	"go.opentelemetry.io/contrib/instrumentation/google.golang.org/grpc/otelgrpc"
)

server := grpc.NewServer(
    grpc.StatsHandler(otelgrpc.NewServerHandler()),
)

Metadata and baggage

On the client side, you can use gRPC metadata as a baggageopen in new window:

import "google.golang.org/grpc/metadata"

md := metadata.Pairs(
	"key1", "value1",
	"key2", "value2",
)
ctx := metadata.NewOutgoingContext(context.Background(), md)

On the server side, you can extract gRPC metadata from the incoming request:

import "google.golang.org/grpc/metadata"

md, ok := metadata.FromIncomingContext(ctx); ok {
    fmt.Println(md)
}

Or use OpenTelemetry baggage API:

import "go.opentelemetry.io/otel/baggage"

baggage := baggage.FromContext(ctx)

What is Uptrace?

Uptrace is a OpenTelemetry backendopen in new window that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace Overview

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demoopen in new window (no login required) or running it locally with Dockeropen in new window. The source code is available on GitHubopen in new window.

What's next?

Next, instrument more operations to get a more detailed picture. Try to prioritize network calls, disk operations, database queries, error and logs.

You can also create your own instrumentations using OpenTelemetry Go Tracing APIopen in new window.

Last Updated: