Ingesting logs using Vector
Vector collects, transforms, and sends your logs to multiple destinations including Uptrace. It is blazingly fast and memory efficient.
Configuration
To configure Vector to send logs to Uptrace, use the HTTP sink and pass your project DSN via HTTP headers.
For example, to collect syslog messages, you can create the following Vector config:
[sources.syslog]
type = "file"
include = ["/var/log/syslog"]
[transforms.parse_syslog]
type = "remap"
inputs = ["syslog"]
source = '''
. = parse_syslog!(string!(.message))
'''
[sinks.uptrace]
type = "http"
method = "post"
inputs = ["parse_syslog"]
encoding.codec = "json"
framing.method = "newline_delimited"
compression = "gzip"
request.headers.uptrace-dsn = "https://<token>@api.uptrace.dev/<project_id>"
uri = "https://api.uptrace.dev/api/v1/vector/logs"
[sources.syslog]
type = "file"
include = ["/var/log/syslog"]
[transforms.parse_syslog]
type = "remap"
inputs = ["syslog"]
source = '''
. = parse_syslog!(string!(.message))
'''
[sinks.uptrace]
type = "http"
method = "post"
inputs = ["parse_syslog"]
encoding.codec = "json"
framing.method = "newline_delimited"
compression = "gzip"
request.headers.uptrace-dsn = "http://project2_secret_token@localhost:14317/2"
uri = "http://localhost:14318/api/v1/vector/logs"
Copy the config above to vector.toml
and then start Vector:
vector --config=vector.toml
To see the data Vector sends to Uptrace, use console sink:
[sinks.my_sink_id]
type = "console"
inputs = [ "my-source-or-transform-id" ]
See vector-logs example for details.
Converting logs to spans
When parsing HTTP or SQL logs, it might be useful to convert logs into spans by providing span name, duration, and some other fields.
For example, you can set attributes using remap
transformation:
[transforms.span_attrs]
type = "remap"
inputs = ["in"]
source = '''
.span_name = "<span name>"
.span_event_name = "span"
.span_kind = "server"
.span_duration = 12345 # nanoseconds
'''
Uptrace recognizes the following span-related attributes:
trace_id
in hex-encoded format, for example,958180131ddde684c1dbda1aeacf51d3
.span_id
is the id of the parent span, for example,0cf859e4f7510204
.span_name
. Optional.span_event_name
must be set tospan
.span_kind
.span_duration
in nanoseconds.
You can also use all available semantic attributes.
Heroku logs
You can collect logs from Heroku’s Logplex and push them to Uptrace.
Fly logs
You can ship logs from Fly.io apps to Uptrace using NATS and Vector.