Using Vector to ingest logs into Uptrace

Ingest Vector Logs

Vectoropen in new window collects, transforms, and sends your logs to multiple destinations including Uptrace. It is blazingly fast and memory efficient.

Configuration

To configure Vector to send logs to Uptrace, use the HTTP sink and pass your project DSN via HTTP headers.

For example, to collect syslog messages, you can create the following Vector config:

[sources.syslog]
type = "file"
include = ["/var/log/syslog"]

[transforms.parse_syslog]
type = "remap"
inputs = ["syslog"]
source = '''
. = parse_syslog!(string!(.message))
'''

[sinks.uptrace]
type = "http"
method = "post"
inputs = ["parse_syslog"]
encoding.codec = "json"
framing.method = "newline_delimited"
compression = "gzip"
request.headers.uptrace-dsn = "https://token@api.uptrace.dev/project_id"
uri = "https://api.uptrace.dev/api/v1/vector/logs"

[sources.syslog]
type = "file"
include = ["/var/log/syslog"]

[transforms.parse_syslog]
type = "remap"
inputs = ["syslog"]
source = '''
. = parse_syslog!(string!(.message))
'''

[sinks.uptrace]
type = "http"
method = "post"
inputs = ["parse_syslog"]
encoding.codec = "json"
framing.method = "newline_delimited"
compression = "gzip"
request.headers.uptrace-dsn = "http://project2_secret_token@localhost:14317/2"
uri = "http://localhost:14318/api/v1/vector/logs"

Copy the config above to vector.toml and then start Vector:

vector --config=vector.toml

To see the data Vector sends to Uptrace, use the consoleopen in new window sink:

[sinks.my_sink_id]
type = "console"
inputs = [ "my-source-or-transform-id" ]

See vector-logsopen in new window example for details.

VRL and attributes

Vector remap language (VRL) allows you to parse logs and set key-value pairs (attributes), for example:

[transforms.parse_apache]
type = "remap"
inputs = ["apache_common_logs"]
source = '''
. = parse_apache_log!(string!(.message), "common")
.log.source = "apache"
'''

Because the log.source attribute name contains a dot, the example above will produce the following nested JSON:

{
  "log": {
    "source": "apache"
  }
}

You can fix that by quoting the attribute key like this:

source = '''
. = parse_apache_log!(string!(.message), "common")
."log.source" = "apache"
'''

Which will produce a flat JSON structure:

{
  "log.source": "apache"
}

Display name and grouping

Typically Uptrace is able to automatically generate a short summary for logs using the log.message attribute, but occasionally you may want to provide a custom summary.

This is where the display.name attribute comes in handy. The display.name attribute is a human-readable string that provides a short summary of the log event.

[transforms.parse_apache]
type = "remap"
inputs = ["apache_common_logs"]
source = '''
. = parse_apache_log!(string!(.message), "common")
."display.name" = join([.protocol, .method], " ") ?? ""
'''

Uptrace does not use display names for grouping, so you're free to put whatever you want there.

To control how Uptrace groups logs together, you can specify the grouping.fingerprint attribute which can be a string or a number (hash). Uptrace will group logs with the same fingerprint together.

[transforms.parse_apache]
type = "remap"
inputs = ["apache_common_logs"]
source = '''
. = parse_apache_log!(string!(.message), "common")
."grouping.fingerprint" = join([.protocol, .method], " ") ?? ""
'''

Converting logs to spans

When parsing HTTP or SQL logs, it might be useful to convert logs into spans by providing span name, duration, and some other fields.

For example, you can set attributes using remap transformation:

[transforms.span_attrs]
type = "remap"
inputs = ["in"]
source = '''
.span_name = "<span name>"
.span_kind = "server"
.span_duration = 12345 # nanoseconds
'''

Uptrace recognizes the following span-related attributes:

trace_id in hex-encoded format, for example, 958180131ddde684c1dbda1aeacf51d3.
span_id is the id of the parent span, for example, 0cf859e4f7510204.
span_name is the span name. Required.
span_kind is the span kind.
span_duration is the span duration in nanoseconds.

You can also use all available semantic attributesopen in new window.

Heroku logs

You can collect logs from Heroku’s Logplex and push them to Uptrace.

Fly logs

You can ship logs from Fly.io apps to Uptrace using NATS and Vectoropen in new window.

# Using Vector to ingest logs into Uptrace

# Configuration

# VRL and attributes

# Display name and grouping

# Converting logs to spans

# Heroku logs

# Fly logs

# See also