OpenTelemetry Profiles: The Fourth Signal

Profiles are the fourth OpenTelemetry signal, after traces, metrics, and logs. A profile is a stream of sampled stack traces showing where a process spends CPU time or allocates memory — the same data a profiler like pprof or perf produces, carried over OTLP alongside the other signals.

The signal answers a question the other three cannot. Metrics say the service is slow, traces say which span is slow, logs say what the code reported while it happened — and profiles say which functions were executing on the CPU during that span.

Status

The next milestone is Beta. No GA date has been committed.

This is a different shape of rollout from the earlier signals. Traces and metrics arrived as SDK APIs first and collection agents later; profiles arrived as a whole-system agent first, with SDK APIs still ahead.

What a profile contains

The data model is built around deduplicated stack traces. A naive profile repeats the same function names in thousands of samples; the OTLP representation stores each location once and has samples reference it, which is where the reported ~40% reduction in wire size compared to raw pprof comes from.

Each sample carries:

• A stack trace — the chain of function calls executing when the sample was taken
• A value — CPU time, allocated bytes, or another measured quantity
• Attributes — the same key-value pairs used by the other signals
• trace_id and span_id — the span that was active when the sample was captured, when one was

That last field is the reason profiles belong in OpenTelemetry rather than in a separate tool. It makes the link between a slow span and the functions running during it a lookup rather than a guess based on timestamps.

The format converts to and from pprof without data loss, so existing pprof tooling keeps working.

Collecting profiles today

Because there are no SDK APIs yet, profiles come from an external agent. The reference implementation is opentelemetry-ebpf-profiler, contributed by Elastic.

It is a whole-system profiler: it samples every process on the host from the kernel, rather than being linked into one application. That means no code changes and no per-language agent, at the cost of requiring privileges and a recent kernel.

Requirements

• Linux, kernel 5.10 or newer
• Root or the equivalent capabilities, since it loads eBPF programs

Runtime coverage

Native code — C, C++, Rust, Go, Zig — is unwound without debug symbols on the host. Interpreted and JIT runtimes are supported individually: HotSpot JVM, Python, Ruby, PHP, Node.js and V8, Perl, Erlang/BEAM, and .NET.

Running it

Standalone, sending to a collection agent:

sudo ./ebpf-profiler -collection-agent=127.0.0.1:11000 -disable-tls

Or as a Collector receiver, which requires the profiles feature gate:

sudo ./otelcol-ebpf-profiler \
  --feature-gates=+service.profilesSupport \
  --config local.example.yaml

The local.example.yaml in the profiler repository under cmd/otelcol-ebpf-profiler/ is the current reference for a working profiles pipeline. Profiles pipeline configuration is still changing between Collector releases, so copy from that file rather than from a tutorial.

Collector components

Profiles reuse the Collector's existing pipeline model — receivers, processors, exporters — with a profiles pipeline type alongside traces, metrics, and logs.

Available today:

• Receivers — the eBPF profiler as a Collector receiver, and a pprof receiver for ingesting profile files
• Processors — k8sattributes for attaching pod and namespace metadata, and OTTL in the transform processor for custom rules, configured like any other Collector pipeline
• Exporters — OTLP, to any backend that accepts the profiles signal

The processor support matters more than it sounds. Because profiles carry the same resource and attribute model as the other signals, k8sattributes labels a profile with the same k8s.pod.name it puts on the traces from that pod, and the two join on those attributes.

What is not ready

Being explicit about the gaps, since the signal is easy to over-plan around:

• No SDK APIs. You cannot start or stop profiling from application code, or attach application-level attributes at the point of capture. Everything comes from the external agent.
• Linux only. The eBPF profiler does not run on macOS or Windows.
• Thin backend support. The specification ships the wire format; storing and querying profiles is up to backends, and most are still building it. Uptrace does not ingest profiles yet.
• The model can still change. Alpha means breaking changes are permitted between releases.

How profiles relate to the other signals

The overlap with tracing is worth being precise about. A span tells you that checkout.processPayment took 400ms. It does not tell you whether that time went to JSON serialization, TLS handshakes, or garbage collection — the span would have to be instrumented at that granularity, which nobody does. A profile covering the same interval shows the function-level breakdown without any instrumentation at all.

They are complementary rather than alternatives: tracing gives you request-scoped causality, profiling gives you code-level cost.

What's next?

• OpenTelemetry distributed tracing — the signal profiles correlate with
• OpenTelemetry Collector — the pipeline profiles flow through
• OpenTelemetry architecture — how signals, SDK, and Collector fit together
• OpenTelemetry metrics — the signal that tells you something is wrong before you profile
• OpenTelemetry troubleshooting — when a profiles pipeline produces nothing