OpenTelemetry Logrus logs [otellogrus]

OpenTelemetry Logrus is a plugin for Logrus logging library that provides integration with OpenTelemetry. Logrus is a popular logging library for Go programming language that provides a simple and easy-to-use API for logging messages in your applications.

OpenTelemetry Logrus automatically injects trace and span IDs for each logging statement, allowing you to correlate your logs with other telemetry data from your application.

Quick Setup

Minimal working example:

import (
    "github.com/sirupsen/logrus"
    "github.com/uptrace/opentelemetry-go-extra/otellogrus"
)

func main() {
    // Add OpenTelemetry hook to Logrus
    logrus.AddHook(otellogrus.NewHook())

    // Your application code...
}

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize and simplify the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry supports multiple programming languages and platforms, making it suitable for a wide range of applications and environments.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization.

How Log Correlation Works

When you use logrus.WithContext(ctx), the otellogrus hook extracts trace information from the context and adds it to your log entries. This creates a connection between your logs and distributed traces, enabling you to:

• Jump from a log message directly to the related trace
• See all logs associated with a specific request or transaction
• Correlate errors in logs with the spans where they occurred

The hook adds these fields to log entries:

Logrus instrumentation

To install otellogrus instrumentation:

go get github.com/uptrace/opentelemetry-go-extra/otellogrus

• Example
• Reference

Usage

You need to install an otellogrus.Hook and use logrus.WithContext to propagate the active span.

import (
    "context"
    "errors"

    "github.com/sirupsen/logrus"
    "github.com/uptrace/opentelemetry-go-extra/otellogrus"
    "go.opentelemetry.io/otel"
)

func main() {
    // Instrument logrus with specific log levels
    logrus.AddHook(otellogrus.NewHook(otellogrus.WithLevels(
        logrus.PanicLevel,
        logrus.FatalLevel,
        logrus.ErrorLevel,
        logrus.WarnLevel,
    )))

    // Create a span for demonstration
    tracer := otel.Tracer("my-service")
    ctx, span := tracer.Start(context.Background(), "my-operation")
    defer span.End()

    // Use ctx to pass the active span to logrus
    logrus.WithContext(ctx).
        WithError(errors.New("hello world")).
        WithField("foo", "bar").
        Error("something failed")
}

Example Log Output

With JSON formatter, the log output includes trace context:

{
  "level": "error",
  "msg": "something failed",
  "time": "2024-01-15T10:30:00Z",
  "foo": "bar",
  "error": "hello world",
  "trace_id": "abc123def456789",
  "span_id": "1234567890abcdef"
}

Options

otellogrus.NewHook accepts the following options:

Configuring Log Levels

By default, the hook fires on all log levels. You can restrict it to specific levels:

// Only record error-level and above logs in traces
logrus.AddHook(otellogrus.NewHook(otellogrus.WithLevels(
    logrus.PanicLevel,
    logrus.FatalLevel,
    logrus.ErrorLevel,
)))

Setting Error Status on Spans

Configure when a span should be marked with an error status:

// Mark span as error for ErrorLevel and above
logrus.AddHook(otellogrus.NewHook(
    otellogrus.WithLevels(
        logrus.PanicLevel,
        logrus.FatalLevel,
        logrus.ErrorLevel,
        logrus.WarnLevel,
    ),
    otellogrus.WithErrorStatusLevel(logrus.ErrorLevel),
))

HTTP Handler Example

Here's a complete example showing log correlation in an HTTP handler:

package main

import (
    "net/http"

    "github.com/sirupsen/logrus"
    "github.com/uptrace/opentelemetry-go-extra/otellogrus"
    "go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp"
)

func main() {
    // Setup OpenTelemetry (SDK initialization omitted for brevity)

    // Add OpenTelemetry hook to logrus
    logrus.AddHook(otellogrus.NewHook(otellogrus.WithLevels(
        logrus.PanicLevel,
        logrus.FatalLevel,
        logrus.ErrorLevel,
        logrus.WarnLevel,
        logrus.InfoLevel,
    )))

    // Create HTTP handler with tracing
    handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
        ctx := r.Context()

        // Log with trace context
        logrus.WithContext(ctx).
            WithField("method", r.Method).
            WithField("path", r.URL.Path).
            Info("handling request")

        // Process request...

        logrus.WithContext(ctx).Info("request completed")
        w.WriteHeader(http.StatusOK)
    })

    // Wrap with OpenTelemetry HTTP instrumentation
    wrappedHandler := otelhttp.NewHandler(handler, "my-handler")

    http.ListenAndServe(":8080", wrappedHandler)
}

Using with Gin Framework

Example of using otellogrus with the Gin web framework:

import (
    "github.com/gin-gonic/gin"
    "github.com/sirupsen/logrus"
    "github.com/uptrace/opentelemetry-go-extra/otellogrus"
    "go.opentelemetry.io/contrib/instrumentation/github.com/gin-gonic/gin/otelgin"
)

func main() {
    logrus.AddHook(otellogrus.NewHook())

    router := gin.Default()
    router.Use(otelgin.Middleware("my-service"))

    router.GET("/users/:id", func(c *gin.Context) {
        // Context from Gin contains trace information
        logrus.WithContext(c.Request.Context()).
            WithField("user_id", c.Param("id")).
            Info("fetching user")

        // Handle request...
    })

    router.Run(":8080")
}

What is Uptrace?

Uptrace is an OpenTelemetry APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

FAQ

What fields are added to log entries? The otellogrus hook adds trace_id, span_id, and trace_flags fields to log entries when a valid trace context is present.

Does otellogrus work with custom formatters? Yes, otellogrus adds fields to the log entry which work with any Logrus formatter (JSON, text, or custom formatters).

What happens when there's no active span? When logrus.WithContext(ctx) is called without an active span in the context, the hook simply doesn't add trace fields. The log entry is still processed normally.

Can I use otellogrus with Logrus globally? Yes, you can configure Logrus globally with the hook. Just remember to always use WithContext(ctx) when logging to include trace context.

How does otellogrus affect performance? The performance impact is minimal. The hook only extracts trace information from the context, which is a lightweight operation.

Should I include all log levels? For production, consider including only important levels (Error, Warn, Info) to reduce noise. Debug and Trace levels may generate too many events in high-traffic applications.

What's next?

With Logrus instrumentation, your logs are now correlated with distributed traces, making debugging significantly easier.

Next steps to enhance your observability:

• Learn about the OpenTelemetry Go Tracing API for creating custom spans
• Set up the OpenTelemetry Collector for production log processing
• Configure structured logging with JSON format for better parsing
• For alternative logging libraries, explore Zap for high-performance logging or slog for the standard library approach