OpenTelemetry Logs [complete guide]

OpenTelemetry Logs allow to record and collect logs in a way that enables correlations and better integration with other observability signals.

Logs are essential for understanding application behavior, diagnosing issues, and monitoring system health. While distributed tracing and metrics provide valuable insights into system performance, logs offer detailed context and information about specific events, errors, and application behavior.

Overview

OpenTelemetry is an open source observability framework that provides standardized APIs, libraries, and instrumentation for collecting telemetry from applications and systems.

While OpenTelemetry is primarily focused on collecting metrics and traces, it also supports log collection through its logging API. OpenTelemetry embraces existing logging solutions and ensures that it works well with existing logging libraries and log collection tools.

OpenTelemetry provides a logging API that allows you to instrument your applications and generate structured logs. The OpenTelemetry Logging API is designed to work with other telemetry data, such as metrics and traces, to provide a unified observability solution.

OpenTelemetry emphasizes structured logging, allowing you to attach additional contextual information to log entries through attributes or metadata. This allows you to include relevant details such as timestamps, request IDs, user IDs, correlation IDs, and other custom context that can aid in log analysis and troubleshooting.

Different types of logs

OpenTelemetry supports capturing logs from various sources within an application or system. Depending on how logs are generated and collected, logs can be grouped into 3 categories.

System and infrastructure logs

System logs provide valuable information about system operation, performance, and security. System logs are typically generated by various components within the system, including the operating system, applications, network devices, and servers.

System logs are written at the host level and have a predefined format and content that can't be easily changed. System logs don't include information about the trace context.

Legacy first-party logs

First-party logs are generated by in-house applications and record specific application events, errors, and user activities. These logs are useful for application debugging and troubleshooting.

Typically, developers can modify these applications to change how logs are written and what information is included. For example, to correlate logs with traces, developers can manually add the trace context to each log statement or do it automatically using a plugin for their logging library.

For example, to propagate context and associate a log record with a span, you can use the following attributes in the log message:

• trace_id for TraceId, hex-encoded.
• span_id for SpanId, hex-encoded.
• trace_flags for trace flags, formatted according to W3C traceflags format.

For example:

request failed trace_id=958180131ddde684c1dbda1aeacf51d3 span_id=0cf859e4f7510204

New first-party logs

When starting a new project, you can follow OpenTelemetry's recommendations and best practices about how to emit logs using auto-instrumentation or configuring your logging library to use an OpenTelemetry log appender.

OpenTelemetry's logging API allows developers to instrument their applications to produce structured logs that can be collected and processed by logging backends or log management systems. The logging API provides a way to attach additional contextual information to log entries, such as tags, attributes, or metadata.

Use the Logger API to log events or messages at different severity levels, such as debug, info, warn, error, and so on. You can also attach additional attributes or context to the log entries to provide more information.

OpenTelemetry also provides a standardized approach to propagating context within logs across distributed systems. This ensures that the relevant execution context is consistently captured and preserved, even when logs are generated by different components of the system.

OpenTelemetry Logs data model allows to include TraceId and SpanId directly in LogRecords.

OpenTelemetry Collector

OpenTelemetry Collector is a flexible and scalable agent for collecting, processing, and exporting telemetry data. It simplifies the task of receiving and managing telemetry data from multiple sources and enables the export of data to multiple backends or observability systems.

OpenTelemetry Collector supports multiple log sources, including application logs, log files, logging libraries, and third-party logging systems. It provides integrations with popular logging frameworks and libraries, enabling seamless ingestion of log data.

Collector provides the ability to transform and enrich log data. You can modify log attributes, add metadata, or enrich logs with additional contextual information to enhance their value and make them more meaningful for analysis and troubleshooting.

Once collected and processed, OpenTelemetry Collector can export log data to various logging backends or systems. It supports exporting logs to popular logging platforms, storage systems, or log management tools for long-term storage, analysis, and visualization.

Examples

Tailing a simple json file

To collect and send JSON logs to Uptrace, add the following to your OpenTelemetry Collector configuration file:

receivers:
  filelog:
    include: [/var/log/myservice/*.json]
    operators:
      - type: json_parser
        timestamp:
          parse_from: attributes.time
          layout: '%Y-%m-%d %H:%M:%S'

processors:
  batch:

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: '<FIXME>'

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [otlp/uptrace]

See filelogreceiver for details.

Syslog

To collect and send syslog logs to Uptrace, add the following to your OpenTelemetry Collector configuration file:

receivers:
  syslog:
    tcp:
      listen_address: '0.0.0.0:54527'
    protocol: rfc3164
    location: UTC # specify server timezone here
    operators:
      - type: move
        from: attributes.message
        to: body

processors:
  batch:

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: '<FIXME>'

service:
  pipelines:
    logs:
      receivers: [syslog]
      processors: [batch]
      exporters: [otlp/uptrace]

Then you need to configure Rsyslog to forward logs to the OpenTelemetry Collector. You can achieve that by adding the following to the end of /etc/rsyslog.conf file:

*.* action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp"
           action.resumeRetryCount="10"
           queue.type="linkedList" queue.size="10000")

Lastly, restart the Rsyslog service:

sudo systemctl restart rsyslog.service

See syslogreceiver for details.

Kubernetes Logs

To collect and send Kubernetes logs to Uptrace, add the following to your OpenTelemetry Collector configuration file:

receivers:
  filelog:
    include:
      - /var/log/pods/*/*/*.log
    include_file_name: false
    include_file_path: true
    start_at: beginning
    operators:
      - id: get-format
        routes:
          - expr: body matches "^\\{"
            output: parser-docker
          - expr: body matches "^[^ Z]+ "
            output: parser-crio
          - expr: body matches "^[^ Z]+Z"
            output: parser-containerd
            type: router
      - id: parser-crio
        output: extract_metadata_from_filepath
        regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
        timestamp:
          layout: 2006-01-02T15:04:05.999999999Z07:00
          layout_type: gotime
          parse_from: attributes.time
          type: regex_parser
      - id: parser-containerd
        output: extract_metadata_from_filepath
        regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
        timestamp:
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
          parse_from: attributes.time
          type: regex_parser
      - id: parser-docker
        output: extract_metadata_from_filepath
        timestamp:
          layout: '%Y-%m-%dT%H:%M:%S.%LZ'
          parse_from: attributes.time
          type: json_parser
      - id: extract_metadata_from_filepath
        parse_from: attributes["log.file.path"]
        regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
        type: regex_parser
      - from: attributes.stream
        to: attributes["log.iostream"]
        type: move
      - from: attributes.container_name
        to: resource["k8s.container.name"]
        type: move
      - from: attributes.namespace
        to: resource["k8s.namespace.name"]
        type: move
      - from: attributes.pod_name
        to: resource["k8s.pod.name"]
        type: move
      - from: attributes.restart_count
        to: resource["k8s.container.restart_count"]
        type: move
      - from: attributes.uid
        to: resource["k8s.pod.uid"]
        type: move
      - from: attributes.log
        to: body
        type: move

processors:
  batch:

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: '<FIXME>'

service:
  pipelines:
    logs:
      receivers: [filelog]
      processors: [batch]
      exporters: [otlp/uptrace]

See filelogreceiver for details.

Kubernetes Events

To collect and send Kubernetes events to Uptrace, add the following to your OpenTelemetry Collector configuration file:

receivers:
  k8s_events:
    auth_type: serviceAccount

processors:
  batch:

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: '<FIXME>'

service:
  pipelines:
    logs:
      receivers: [k8s_events]
      processors: [batch]
      exporters: [otlp/uptrace]

See k8seventsreceiver for details.

Golang Slog

OpenTelemetry Slog is a bridge between the OpenTelemetry observability framework and the popular slog logging library in Go. It allows you to integrate your existing slog logs with OpenTelemetry traces and metrics, providing a unified view of your application's behavior.

The simplest OpenTelemetry Slog configuration looks like this:

import (
    "go.opentelemetry.io/otel/sdk/log"
    "go.opentelemetry.io/contrib/bridges/otelslog"
)

exp, err := stdoutlog.New()
if err != nil {
    panic(err)
}

processor := log.NewSimpleProcessor(exp)
provider := log.NewLoggerProvider(log.WithProcessor(processor))
defer provider.Shutdown(context.Background())

global.SetLoggerProvider(provider)

logger := otelslog.NewLogger("app_or_package_name")
logger.ErrorContext(ctx, "hello world", slog.String("error", "error message"))

See GitHub example for details.

OpenTelemetry Backend

Once the log data is exported to your logging backend, you can process and analyze the logs using the platform's features. This can include filtering, searching, aggregating, and visualizing the logs to gain insight into your application's behavior and troubleshoot issues.

See OpenTelemetry backend for the list of compatible backends.

Conclusion

Logs are an essential part of observability, and OpenTelemetry can work alongside logging frameworks and libraries to provide a comprehensive observability solution.

• Golang logging
• otelzap