OpenTelemetry Logs
OpenTelemetry Logs allow to record and collect logs in a way that enables correlations and better integration with other observability signals.
Logs are essential for understanding application behavior, diagnosing issues, and monitoring system health. While distributed tracing and metrics provide valuable insights into system performance, logs offer detailed context and information about specific events, errors, and application behavior.
Overview
OpenTelemetry is an open source observability framework that provides standardized APIs, libraries, and instrumentation for collecting telemetry from applications and systems.
While OpenTelemetry is primarily focused on collecting metrics and traces, it also supports log collection through its logging API. OpenTelemetry embraces existing logging solutions and ensures that it works well with existing logging libraries and log collection tools.
OpenTelemetry provides a logging API that allows you to instrument your applications and generate structured logs. The OpenTelemetry Logging API is designed to work with other telemetry data, such as metrics and traces, to provide a unified observability solution.
OpenTelemetry emphasizes structured logging, allowing you to attach additional contextual information to log entries through attributes or metadata. This allows you to include relevant details such as timestamps, request IDs, user IDs, correlation IDs, and other custom context that can aid in log analysis and troubleshooting.
Different types of logs
OpenTelemetry supports capturing logs from various sources within an application or system. Depending on how logs are generated and collected, logs can be grouped into 3 categories.
System and infrastructure logs
System logs provide valuable information about system operation, performance, and security. System logs are typically generated by various components within the system, including the operating system, applications, network devices, and servers.
System logs are written at the host level and have a predefined format and content that can't be easily changed. System logs don't include information about the trace context.
Legacy first-party logs
First-party logs are generated by in-house applications and record specific application events, errors, and user activities. These logs are useful for application debugging and troubleshooting.
Typically, developers can modify these applications to change how logs are written and what information is included. For example, to correlate logs with traces, developers can manually add the trace context to each log statement or do it automatically using a plugin for their logging library.
For example, to propagate context and associate a log record with a span, you can use the following attributes in the log message:
trace_idfor TraceId, hex-encoded.span_idfor SpanId, hex-encoded.trace_flagsfor trace flags, formatted according to W3C traceflags format.
For example:
request failed trace_id=958180131ddde684c1dbda1aeacf51d3 span_id=0cf859e4f7510204
New first-party logs
When starting a new project, you can follow OpenTelemetry's recommendations and best practices about how to emit logs using auto-instrumentation or configuring your logging library to use an OpenTelemetry log appender.
OpenTelemetry's logging API allows developers to instrument their applications to produce structured logs that can be collected and processed by logging backends or log management systems. The logging API provides a way to attach additional contextual information to log entries, such as tags, attributes, or metadata.
Use the Logger API to log events or messages at different severity levels, such as debug, info, warn, error, and so on. You can also attach additional attributes or context to the log entries to provide more information.
OpenTelemetry also provides a standardized approach to propagating context within logs across distributed systems. This ensures that the relevant execution context is consistently captured and preserved, even when logs are generated by different components of the system.
OpenTelemetry Logs data model allows to include TraceId and SpanId directly in LogRecords.
OpenTelemetry Collector
OpenTelemetry Collector is a flexible and scalable agent for collecting, processing, and exporting telemetry data. It simplifies the task of receiving and managing telemetry data from multiple sources and enables the export of data to multiple backends or observability systems.
OpenTelemetry Collector supports multiple log sources, including application logs, log files, logging libraries, and third-party logging systems. It provides integrations with popular logging frameworks and libraries, enabling seamless ingestion of log data.
Collector provides the ability to transform and enrich log data. You can modify log attributes, add metadata, or enrich logs with additional contextual information to enhance their value and make them more meaningful for analysis and troubleshooting.
Once collected and processed, OpenTelemetry Collector can export log data to various logging backends or systems. It supports exporting logs to popular logging platforms, storage systems, or log management tools for long-term storage, analysis, and visualization.
Examples
Tailing a simple json file
To collect and send JSON logs to Uptrace, add the following to your OpenTelemetry Collector configuration file:
receivers:
filelog:
include: [/var/log/myservice/*.json]
operators:
- type: json_parser
timestamp:
parse_from: attributes.time
layout: '%Y-%m-%d %H:%M:%S'
processors:
batch:
exporters:
otlp/uptrace:
endpoint: otlp.uptrace.dev:4317
headers:
uptrace-dsn: 'https://<token>@api.uptrace.dev/<project_id>'
service:
pipelines:
logs:
receivers: [filelog]
processors: [batch]
exporters: [otlp/uptrace]
See filelogreceiver for details.
Syslog
To collect and send syslog logs to Uptrace, add the following to your OpenTelemetry Collector configuration file:
receivers:
syslog:
tcp:
listen_address: '0.0.0.0:54527'
protocol: rfc3164
location: UTC # specify server timezone here
operators:
- type: move
from: attributes.message
to: body
processors:
batch:
exporters:
otlp/uptrace:
endpoint: otlp.uptrace.dev:4317
headers:
uptrace-dsn: 'https://<token>@api.uptrace.dev/<project_id>'
service:
pipelines:
logs:
receivers: [syslog]
processors: [batch]
exporters: [otlp/uptrace]
Then you need to configure Rsyslog to forward logs to the OpenTelemetry Collector. You can achieve that by adding the following to the end of /etc/rsyslog.conf file:
*.* action(type="omfwd" target="0.0.0.0" port="54527" protocol="tcp"
action.resumeRetryCount="10"
queue.type="linkedList" queue.size="10000")
Lastly, restart the Rsyslog service:
sudo systemctl restart rsyslog.service
See syslogreceiver for details.
Kubernetes Logs
To collect and send Kubernetes logs to Uptrace, add the following to your OpenTelemetry Collector configuration file:
receivers:
filelog:
include:
- /var/log/pods/*/*/*.log
include_file_name: false
include_file_path: true
start_at: beginning
operators:
- id: get-format
routes:
- expr: body matches "^\\{"
output: parser-docker
- expr: body matches "^[^ Z]+ "
output: parser-crio
- expr: body matches "^[^ Z]+Z"
output: parser-containerd
type: router
- id: parser-crio
output: extract_metadata_from_filepath
regex: ^(?P<time>[^ Z]+) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
timestamp:
layout: 2006-01-02T15:04:05.999999999Z07:00
layout_type: gotime
parse_from: attributes.time
type: regex_parser
- id: parser-containerd
output: extract_metadata_from_filepath
regex: ^(?P<time>[^ ^Z]+Z) (?P<stream>stdout|stderr) (?P<logtag>[^ ]*) ?(?P<log>.*)$
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
type: regex_parser
- id: parser-docker
output: extract_metadata_from_filepath
timestamp:
layout: '%Y-%m-%dT%H:%M:%S.%LZ'
parse_from: attributes.time
type: json_parser
- id: extract_metadata_from_filepath
parse_from: attributes["log.file.path"]
regex: ^.*\/(?P<namespace>[^_]+)_(?P<pod_name>[^_]+)_(?P<uid>[a-f0-9\-]+)\/(?P<container_name>[^\._]+)\/(?P<restart_count>\d+)\.log$
type: regex_parser
- from: attributes.stream
to: attributes["log.iostream"]
type: move
- from: attributes.container_name
to: resource["k8s.container.name"]
type: move
- from: attributes.namespace
to: resource["k8s.namespace.name"]
type: move
- from: attributes.pod_name
to: resource["k8s.pod.name"]
type: move
- from: attributes.restart_count
to: resource["k8s.container.restart_count"]
type: move
- from: attributes.uid
to: resource["k8s.pod.uid"]
type: move
- from: attributes.log
to: body
type: move
processors:
batch:
exporters:
otlp/uptrace:
endpoint: otlp.uptrace.dev:4317
headers:
uptrace-dsn: 'https://<token>@api.uptrace.dev/<project_id>'
service:
pipelines:
logs:
receivers: [filelog]
processors: [batch]
exporters: [otlp/uptrace]
See filelogreceiver for details.
Kubernetes Events
To collect and send Kubernetes events to Uptrace, add the following to your OpenTelemetry Collector configuration file:
receivers:
k8s_events:
auth_type: serviceAccount
processors:
batch:
exporters:
otlp/uptrace:
endpoint: otlp.uptrace.dev:4317
headers:
uptrace-dsn: 'https://<token>@api.uptrace.dev/<project_id>'
service:
pipelines:
logs:
receivers: [k8s_events]
processors: [batch]
exporters: [otlp/uptrace]
See k8seventsreceiver for details.
OpenTelemetry Backend
Once the log data is exported to your logging backend, you can process and analyze the logs using the platform's features. This can include filtering, searching, aggregating, and visualizing the logs to gain insight into your application's behavior and troubleshoot issues.
See OpenTelemetry backend for the list of compatible backends.
Conclusion
Logs are an essential part of observability, and OpenTelemetry can work alongside logging frameworks and libraries to provide a comprehensive observability solution.
