Prometheus and OpenTelemetry Collector Integration [Full Guide with Code]

This guide explains how to integrate Prometheus with OpenTelemetry, including collecting Prometheus metrics with the OpenTelemetry Collector and exporting OpenTelemetry metrics to Prometheus. You'll learn both directions of integration with working code examples.

Prometheus and OpenTelemetry Integration Overview

OpenTelemetry and Prometheus represent two powerful approaches to observability, each with their own strengths. While they use different data formats, they can be effectively integrated to create comprehensive monitoring solutions.

Key Integration Points:

• Use OpenTelemetry Collector to collect Prometheus metrics via the Prometheus Receiver
• Export OpenTelemetry metrics to Prometheus using the Prometheus Exporter
• Convert between Prometheus and OpenTelemetry metric formats

What is OpenTelemetry?

OpenTelemetry is an open source and vendor-neutral API for distributed tracing (including logs and errors) and OpenTelemetry metrics. With OpenTelemetry, you can instrument your application once and then add or change OpenTelemetry-compatible backends without changing the instrumentation.

What is Prometheus?

Prometheus is a purpose-built monitoring system that excels at metrics collection and alerting in dynamic environments. Created for cloud-native architectures, it employs a pull-based scraping model to collect time-series data from instrumented targets. Its distinguishing features include a dimensional data model, self-contained TSDB storage, and PromQL—a functional query language that enables sophisticated data analysis without external dependencies.

OpenTelemetry Collector for Prometheus

The OpenTelemetry Collector serves as a critical bridge between Prometheus and OpenTelemetry systems. It can:

1. Pull metrics from Prometheus exporters (acting as a Prometheus server)
2. Export metrics to Prometheus (making OpenTelemetry data available to Prometheus)
3. Transform metrics between the two formats

This flexibility allows you to gradually adopt OpenTelemetry while maintaining compatibility with existing Prometheus infrastructure.

Collecting Prometheus Metrics with OpenTelemetry

The OpenTelemetry Collector can pull metrics from Prometheus endpoints using the Prometheus receiver. In this scenario, the Collector functions as a Prometheus server, scraping metrics from Prometheus exporters and then processing and exporting the collected metrics to configured destinations, for example, Uptrace APM.

Configuration Example:

receivers:
  prometheus_simple:
    collection_interval: 10s
    endpoint: '172.17.0.5:9153'
    metrics_path: '/metrics'
    use_service_account: false
    tls:
      ca_file: '/path/to/ca'
      cert_file: '/path/to/cert'
      key_file: '/path/to/key'
      insecure_skip_verify: true

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      # Copy your project DSN here
      uptrace-dsn: '<FIXME>'

service:
  pipelines:
    metrics:
      receivers: [prometheus_simple]
      exporters: [otlp/uptrace]

With this configuration, the OpenTelemetry Collector will scrape Prometheus metrics and forward them to an OpenTelemetry backend (in this example, Uptrace).

Exporting OpenTelemetry Metrics to Prometheus

To make OpenTelemetry metrics available to Prometheus, you can use the OpenTelemetry Collector Prometheus Exporter. This exporter converts OpenTelemetry metrics to the Prometheus format and exposes them as a Prometheus scrape endpoint.

Configuration Example:

exporters:
  prometheus:
    endpoint: '1.2.3.4:1234'
    namespace: test-space
    const_labels:
      label1: value1
      'another label': spaced value
    send_timestamps: true
    metric_expiration: 180m
    resource_to_telemetry_conversion:
      enabled: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

The exporter allows you to add custom labels and metadata to the exported metrics, which can be useful for organizing and categorizing metrics in Prometheus.

Direct Integration: Sending Metrics from Applications to Prometheus

You can directly instrument your applications with OpenTelemetry and make metrics available to Prometheus without using the OpenTelemetry Collector.

Go Example: OpenTelemetry to Prometheus

Here's how to create OpenTelemetry metrics and expose them for Prometheus scraping in Go:

import "go.opentelemetry.io/otel/metric/instrument"

counter := Meter.SyncInt64().Counter(
    "test.my_counter",
    instrument.WithUnit("1"),
    instrument.WithDescription("Just a test counter"),
)

// Increment the counter.
counter.Add(ctx, 1, attribute.String("foo", "bar"))
counter.Add(ctx, 10, attribute.String("hello", "world"))

To expose these metrics for Prometheus:

import (
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/metric/global"
    "go.opentelemetry.io/otel/sdk/export/metric/aggregation"
    "go.opentelemetry.io/otel/sdk/metric/aggregator/histogram"
    controller "go.opentelemetry.io/otel/sdk/metric/controller/basic"
    processor "go.opentelemetry.io/otel/sdk/metric/processor/basic"
    selector "go.opentelemetry.io/otel/sdk/metric/selector/simple"
)

func configureOpentelemetry() {
    exporter := configureMetrics()

    http.HandleFunc("/metrics", exporter.ServeHTTP)
    fmt.Println("listenening on http://localhost:8088/metrics")

    go func() {
        _ = http.ListenAndServe(":8088", nil)
    }()
}

func configureMetrics() *prometheus.Exporter {
    config := prometheus.Config{}

    ctrl := controller.New(
        processor.NewFactory(
            selector.NewWithHistogramDistribution(
                histogram.WithExplicitBoundaries(config.DefaultHistogramBoundaries),
            ),
            export.CumulativeExportKindSelector(),
            processor.WithMemory(true),
        ),
    )

    exporter, err := prometheus.New(config, ctrl)
    if err != nil {
        panic(err)
    }

    global.SetMeterProvider(exporter.MeterProvider())

    return exporter
}

Configure Prometheus to scrape these exposed metrics:

scrape_configs:
  - job_name: 'opentelemetry'
    static_configs:
      - targets: ['localhost:8088']

A complete runnable example is available on GitHub.

Comparing OpenTelemetry and Prometheus

While both OpenTelemetry and Prometheus are powerful observability tools, they have different strengths and focuses:

For a more detailed comparison, see OpenTelemetry vs Prometheus.

Practical Use Cases for Prometheus OpenTelemetry Integration

1. Migration from Prometheus to OpenTelemetry: Use the Prometheus receiver to collect existing metrics while transitioning to OpenTelemetry.
2. Hybrid Monitoring Systems: Leverage both Prometheus for alerting and OpenTelemetry for distributed tracing.
3. Extended Visualization: Export OpenTelemetry metrics to Prometheus for visualization in Grafana dashboards.
4. Legacy System Integration: Use Prometheus exporters for legacy systems while adopting OpenTelemetry for new applications.

Getting Started with Uptrace

Uptrace is an open source APM that supports both OpenTelemetry and Prometheus metrics. It features:

• Intuitive query builder and rich dashboards
• Alerting capabilities
• Integration with most languages and frameworks
• Efficient storage with ClickHouse database

You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.

Additional Resources

• Jaeger vs Prometheus
• DataDog vs Prometheus
• OpenTelemetry Metrics
• OpenTelemetry Collector
• What is OpenTelemetry?