Monitor AWS Lambda Golang with OpenTelemetry
AWS Lambda is a serverless, event-driven compute service that lets you run code without provisioning or managing servers.
You can use OpenTelemetry with AWS Lambda to enable distributed tracing and observability for serverless applications.
What is OpenTelemetry?
OpenTelemetry defines APIs and protocols for collecting telemetry data such as metrics, traces, and logs, and provides a variety of libraries, agents, and integrations for popular programming languages and technologies.
OpenTelemetry is an open and vendor-neutral solution that provides a unified approach to observability, making it easier for organizations to manage the complexity of their cloud-native infrastructure. It enables organizations to collect telemetry from their applications and send it to a variety of of distributed tracing tools.
AWS Lambda containers
Lambda runs the code in isolated containers dynamically scaling the number of containers as needed. When there are no new requests, Lambda freezes idle containers.
When the container is frozen, all processes in the container will be frozen as well. If the process uses a timer to flush the buffered data, the timer won’t fire until the container thaws. The interval between the frozen and the thaw state is unpredictable, ranging from seconds to hours.
To mitigate that problem, the instrumentation has to flush data out before the Lambda function returns.
Installation
To install OpenTelemetry AWS Lambda instrumentation:
go get -u go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda
Usage
You can instrument your AWS Lambda function with OpenTelemetry like this:
import "go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda"
func main() {
lambda.Start(otellambda.InstrumentHandler(HandleRequest))
}
To use otellambda with OpenTelemetry Go, you need to configure otellambda.WithFlusher with the tracer provider used by Uptrace.
import (
"go.opentelemetry.io/contrib/propagators/aws/xray"
"go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda"
"go.opentelemetry.io/contrib/instrumentation/github.com/aws/aws-lambda-go/otellambda/xrayconfig"
"github.com/uptrace/uptrace-go/uptrace"
)
func main() {
ctx := context.Background()
uptrace.ConfigureOpentelemetry(
// copy your project DSN here or use UPTRACE_DSN env var
//uptrace.WithDSN("https://token@api.uptrace.dev/project_id"),
uptrace.WithServiceName("myservice"),
uptrace.WithServiceVersion("v1.0.0"),
)
defer uptrace.Shutdown(ctx)
tp := uptrace.TracerProvider()
lambda.Start(otellambda.InstrumentHandler(
lambdaHandler(ctx),
// Flush buffered spans to Uptrace.
otellambda.WithFlusher(tp),
))
}
See aws-lambda example for details.
OpenTelemetry Lambda
Alternatively, you can add opentelemetry-lambda as a Lambda layer, but it does not offer any advantages at the moment. See the example for details.
What is Uptrace?
Uptrace is a DataDog alternative that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.
Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.
Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.
In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.
What's next?
Next, instrument more operations to get a more detailed picture. Try to prioritize network calls, disk operations, database queries, error and logs.
You can also create your own instrumentations using OpenTelemetry Go Tracing API.
