Monitor AWS Lambda Golang with OpenTelemetry
AWS Lambda is a serverless, event-driven compute service that lets you run code without provisioning or managing servers.
This article explains how you can monitor and optimize AWS Lambda Golang performance using OpenTelemetry observability framework.
What is OpenTelemetry?
OpenTelemetry is an open source and vendor-neutral API for OpenTelemetry tracing (including logs and errors) and OpenTelemetry metrics.
Otel specifies how to collect and export telemetry data in a vendor agnostic way. With OpenTelemetry, you can instrument your application once and then add or change vendors without changing the instrumentation, for example, many open source tracing tools already support OpenTelemetry.
OpenTelemetry is available for most programming languages and provides interoperability across different languages and environments.
AWS Lambda containers
Lambda runs the code in isolated containers dynamically scaling the number of containers as needed. When there are no new requests, Lambda freezes idle containers.
When the container is frozen, all processes in the container will be frozen as well. If the process uses a timer to flush the buffered data, the timer won’t fire until the container thaws. The interval between the frozen and the thaw state is unpredictable, ranging from seconds to hours.
To mitigate that problem, the instrumentation has to flush data out before the Lambda function returns.
Installation
To install OpenTelemetry AWS Lambda instrumentation:
go get -u go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda
Usage
You can instrument your AWS Lambda function with OpenTelemetry like this:
import "go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda"
func main() {
lambda.Start(otellambda.InstrumentHandler(HandleRequest))
}
To use otellambda with OpenTelemetry Go, you need to configure otellambda.WithFlusher with the tracer provider used by Uptrace.
import (
"go.opentelemetry.io/contrib/propagators/aws/xray"
"go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda"
"go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda/xrayconfig"
"github.com/uptrace/uptrace-go/uptrace"
)
func main() {
ctx := context.Background()
uptrace.ConfigureOpentelemetry(
// copy your project DSN here or use UPTRACE_DSN env var
//uptrace.WithDSN("https://<token>@uptrace.dev/<project_id>"),
uptrace.WithServiceName("myservice"),
uptrace.WithServiceVersion("v1.0.0"),
)
defer uptrace.Shutdown(ctx)
tp := uptrace.TracerProvider()
lambda.Start(otellambda.InstrumentHandler(
lambdaHandler(ctx),
// Flush buffered spans to Uptrace.
otellambda.WithFlusher(tp),
))
}
See aws-lambda example for details.
OpenTelemetry Lambda
Alternatively, you can add opentelemetry-lambda as a Lambda layer, but it does not offer any advantages at the moment. See example for details.
What is Uptrace?
Uptrace is an open source APM for OpenTelemetry with an intuitive query builder, rich dashboards, automatic alerts, and integrations for most languages and frameworks.
You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.
What's next?
Next, instrument more operations, for example, database queries, errors, and logs. You can also learn about OpenTelemetry Go Tracing API to create your own instrumentations.
Popular instrumentations:
