Structured Logging with Uptrace

• Structured logging
  • logfmt
  • JSON format
  • Free format
• Application logs
  • Go
  • Python
• Grouping logs together
• Third-party logs
  • Propagating trace context

Structured logging

In structured logging, log messages are broken down into key-value pairs, making it easier to search, filter, and analyze logs. This is in contrast to traditional logging, which usually consists of unstructured text that is difficult to parse and analyze.

Backends like Uptrace automatically parse such messages and save the extracted structured data as attributes.

logfmt

In logfmt, log message consists of key/value pairs separated with a space. Therefore you must quote values containing spaces.

request failed http.method=GET http.route=/users/:id enduser.id=123 foo="hello world"

JSON format

You can also use JSON to include structured data in your log messages:

request failed {"http.method": "GET", "http.route": "/users/:id", "enduser.id": 123, "foo": "hello world"}

Free format

If your library does not support structured logging, you can still improve grouping by quoting params:

# good
can't parse string: "the original string"

# bad
can't parse string: the original string

Application logs

To record application logs, you can use OpenTelemetry span events. You must set the event name to log and use semantic attributes to record the context:

• log.severity to record the log severity. Must be one of TRACE, DEBUG, INFO, WARN, ERROR, FATAL, and PANIC.
• log.message to record the message.
• code.function to record the function name.
• code.filepath to record the file path.
• code.lineno to record the line number.

Go

For example, using Go programming language and OpenTelemetry events API:

span := trace.SpanFromContext(ctx)

span.AddEvent("log", trace.WithAttributes(
    // Log severity and message.
    attribute.String("log.severity", "ERROR"),
    attribute.String("log.message", "request failed"),

    // Optional.
    attribute.String("code.function", "org.FetchUser"),
    attribute.String("code.filepath", "org/user.go"),
    attribute.Int("code.lineno", 123),

    // Additional details.
    attribute.String("foo", "hello world"),
))

You can also use instrumentations for popular logging libraries which allow recording logs using a more conventional API, for example, OpenTelemetry Zap and OpenTelemetry Logrus.

Python

OpenTelemetry Python comes with a handler for Python's logging package so you can just use the standard logging API:

import logging
import uptrace

# Configure OpenTelemetry.
uptrace.configure_opentelemetry(...)

# Use logging API as usual.
logger = logging.getLogger(__name__)
logger.setLevel(logging.ERROR)

Grouping logs together

You can control how Uptrace groups logs together by providing log.fingerprint attribute which can be a string or a number (hash/id):

log.severity = "info"
log.message = "unstructured log message 123 456 789"
log.fingerprint = "unstructured log message"

Third-party logs

To collect existing third-party logs, for example, syslog or nginx logs, you can use Vector and FluentBit integrations.

Propagating trace context

When using third-party logs, trace context is not automatically propagated and logs can't be linked with spans.

To propagate context and associate a log entry with a span, use the following attribute keys in the log message:

• trace_id for TraceId, hex-encoded.
• span_id for SpanId, hex-encoded.
• trace_flags for trace flags, formatted according to W3C traceflags format.

For example:

request failed trace_id=958180131ddde684c1dbda1aeacf51d3 span_id=0cf859e4f7510204