Structured Logging with Uptrace
Structured logging
In structured logging, log messages are broken down into key-value pairs, making it easier to search, filter, and analyze logs. This is in contrast to traditional logging, which usually consists of unstructured text that is difficult to parse and analyze.
Backends like Uptrace automatically parse such messages and save the extracted structured data as attributes.
logfmt
In logfmt, log message consists of key/value pairs separated with a space. Therefore you must quote values containing spaces.
request failed http.method=GET http.route=/users/:id enduser.id=123 foo="hello world"
JSON format
You can also use JSON to include structured data in your log messages:
request failed {"http.method": "GET", "http.route": "/users/:id", "enduser.id": 123, "foo": "hello world"}
Free format
If your library does not support structured logging, you can still improve grouping by quoting params:
# good
can't parse string: "the original string"
# bad
can't parse string: the original string
Application logs
To record application logs, you can use OpenTelemetry span events. You must set the event name to log
and use semantic attributes to record the context:
log.severity
to record the log severity. Must be one ofTRACE
,DEBUG
,INFO
,WARN
,ERROR
,FATAL
, andPANIC
.log.message
to record the message.code.function
to record the function name.code.filepath
to record the file path.code.lineno
to record the line number.
Go
For example, using Go programming language and OpenTelemetry events API:
span := trace.SpanFromContext(ctx)
span.AddEvent("log", trace.WithAttributes(
// Log severity and message.
attribute.String("log.severity", "ERROR"),
attribute.String("log.message", "request failed"),
// Optional.
attribute.String("code.function", "org.FetchUser"),
attribute.String("code.filepath", "org/user.go"),
attribute.Int("code.lineno", 123),
// Additional details.
attribute.String("foo", "hello world"),
))
You can also use instrumentations for popular logging libraries which allow recording logs using a more conventional API, for example, OpenTelemetry Zap and OpenTelemetry Logrus.
Python
OpenTelemetry Python comes with a handler for Python's logging
package so you can just use the standard logging API:
import logging
import uptrace
# Configure OpenTelemetry.
uptrace.configure_opentelemetry(...)
# Use logging API as usual.
logger = logging.getLogger(__name__)
logger.setLevel(logging.ERROR)
Grouping logs together
You can control how Uptrace groups logs together by providing log.fingerprint
attribute which can be a string or a number (hash/id):
log.severity = "info"
log.message = "unstructured log message 123 456 789"
log.fingerprint = "unstructured log message"
Third-party logs
To collect existing third-party logs, for example, syslog or nginx logs, you can use Vector and FluentBit integrations.
Propagating trace context
When using third-party logs, trace context is not automatically propagated and logs can't be linked with spans.
To propagate context and associate a log entry with a span, use the following attribute keys in the log message:
trace_id
for TraceId, hex-encoded.span_id
for SpanId, hex-encoded.trace_flags
for trace flags, formatted according to W3C traceflags format.
For example:
request failed trace_id=958180131ddde684c1dbda1aeacf51d3 span_id=0cf859e4f7510204