Prometheus and OpenTelemetry Collector Integration [Full Guide with Code]

This guide explains how to integrate Prometheus with OpenTelemetry, including collecting Prometheus metrics with the OpenTelemetry Collector and exporting OpenTelemetry metrics to Prometheus. You'll learn both directions of integration with working code examples.

Quick Reference

Component	Type	Direction	Use Case
Prometheus Receiver	Pull	Prometheus → OTel	Scrape Prometheus endpoints into Collector
Prometheus Exporter	Pull	OTel → Prometheus	Expose OTel metrics for Prometheus scraping
PrometheusRemoteWrite	Push	OTel → Prometheus	Push metrics to Prometheus-compatible backends

Key Configuration Options:

resource_to_telemetry_conversion - Convert resource attributes to metric labels
scrape_configs - Full Prometheus scrape configuration support
endpoint - Where metrics are exposed or sent

Integration Overview

OpenTelemetry and Prometheus represent two powerful approaches to observability, each with their own strengths. While they use different data formats, they can be effectively integrated to create comprehensive monitoring solutions.

Key Integration Points:

Use OpenTelemetry Collector to collect Prometheus metrics via the Prometheus Receiver
Export OpenTelemetry metrics to Prometheus using the Prometheus Exporter
Convert between Prometheus and OpenTelemetry metric formats

What is OpenTelemetry?

OpenTelemetry is an open source and vendor-neutral API for distributed tracing (including logs and errors) and OpenTelemetry metrics. With OpenTelemetry, you can instrument your application once and then add or change OpenTelemetry-compatible backends without changing the instrumentation.

What is Prometheus?

Prometheus is a purpose-built monitoring system that excels at metrics collection and alerting in dynamic environments. Created for cloud-native architectures, it employs a pull-based scraping model to collect time-series data from instrumented targets. Its distinguishing features include a dimensional data model, self-contained TSDB storage, and PromQL—a functional query language that enables sophisticated data analysis without external dependencies.

OpenTelemetry Collector for Prometheus

The OpenTelemetry Collector serves as a critical bridge between Prometheus and OpenTelemetry systems. It can:

Pull metrics from Prometheus exporters (acting as a Prometheus server)
Export metrics to Prometheus (making OpenTelemetry data available to Prometheus)
Transform metrics between the two formats

This flexibility allows you to gradually adopt OpenTelemetry while maintaining compatibility with existing Prometheus infrastructure.

Collecting Prometheus Metrics

The OpenTelemetry Collector can pull metrics from Prometheus endpoints using Prometheus receivers. In this scenario, the Collector functions as a Prometheus server, scraping metrics from Prometheus exporters and then processing and exporting the collected metrics to configured destinations.

💡 Backend Configuration: The examples below use Uptrace as the backend, but you can export to any OTLP-compatible backend (Jaeger, Grafana Cloud, Prometheus itself, etc.) by changing the endpoint and headers configuration.

Receiver Configuration

The Collector provides two receivers for Prometheus metrics:

yaml Simple Receiver (Single Target)

# Use for scraping a single endpoint with minimal configuration
# https://github.com/open-telemetry/opentelemetry-collector-contrib/tree/main/receiver/simpleprometheusreceiver
receivers:
  prometheus_simple:
    collection_interval: 10s
    endpoint: '172.17.0.5:9153'
    metrics_path: '/metrics'
    use_service_account: false
    tls:
      ca_file: '/path/to/ca'
      cert_file: '/path/to/cert'
      key_file: '/path/to/key'
      insecure_skip_verify: true

exporters:
  otlp/uptrace:
    endpoint: api.uptrace.dev:4317
    headers:
      uptrace-dsn: '<FIXME>'

service:
  pipelines:
    metrics:
      receivers: [prometheus_simple]
      exporters: [otlp/uptrace]

Exporting Metrics to Prometheus

To make OpenTelemetry metrics available to Prometheus, you can use the OpenTelemetry Collector Prometheus Exporter. This exporter converts OpenTelemetry metrics to the Prometheus format and exposes them as a Prometheus scrape endpoint.

Prometheus Exporter Configuration

yaml

exporters:
  prometheus:
    endpoint: '0.0.0.0:8889'
    namespace: test-space
    const_labels:
      label1: value1
      'another label': spaced value
    send_timestamps: true
    metric_expiration: 180m
    resource_to_telemetry_conversion:
      enabled: true  # Converts resource attributes to metric labels (see explanation below)

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

After starting the Collector, metrics will be available at http://0.0.0.0:8889/metrics for Prometheus to scrape.

Configure Prometheus to scrape the endpoint:

yaml

scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
      - targets: ['otel-collector:8889']

resource_to_telemetry_conversion

The resource_to_telemetry_conversion setting controls how OpenTelemetry resource attributes are exposed in Prometheus:

enabled: false (default)

Resource attributes stored separately in target_info metric
Requires metric joins in PromQL queries
Lower cardinality, better for high-volume environments

enabled: true

All resource attributes become direct metric labels
Simpler PromQL queries (no joins needed)
Higher cardinality, use with caution

Example with enabled: false:

promql

# Query requires join with target_info
sum(
  http_requests_total * on(job, instance) group_left(service_name, service_version)
  target_info
)

Example with enabled: true:

promql

# Direct query with resource attributes as labels
sum(http_requests_total{service_name="my-app", service_version="1.0.0"})

Recommendation: Use enabled: false (default) for production to avoid cardinality explosion. Only enable when you need simplified queries and understand the cardinality impact.

PrometheusRemoteWrite Exporter

The PrometheusRemoteWrite Exporter provides a push-based alternative to the pull-based Prometheus exporter. It sends metrics directly to Prometheus-compatible backends like Prometheus itself, Cortex, Mimir, or Thanos.

When to Use PrometheusRemoteWrite

Prometheus Exporter: Prometheus scrapes metrics from Collector (pull)
PrometheusRemoteWrite: Collector pushes metrics to Prometheus (push)

Use PrometheusRemoteWrite when:

Prometheus can't reach the Collector (firewall, NAT)
Sending metrics to remote Prometheus backends (Cortex, Mimir, Thanos)
Centralized metric collection from multiple Collectors

Configuration Example

yaml

exporters:
  prometheusremotewrite:
    endpoint: "https://prometheus-server/api/v1/write"
    resource_to_telemetry_conversion:
      enabled: true
    external_labels:
      cluster: "prod"
      region: "us-east-1"

    # Optional: TLS configuration
    tls:
      insecure: false
      cert_file: '/path/to/cert'
      key_file: '/path/to/key'

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheusremotewrite]

Key configuration options:

endpoint - Remote write URL (required)
external_labels - Labels added to all metrics
resource_to_telemetry_conversion - Same behavior as Prometheus exporter
add_metric_suffixes - Add _total, _bucket suffixes (default: true)
send_metadata - Send Prometheus metadata (default: false)

Direct App Integration

You can directly instrument your applications with OpenTelemetry and make metrics available to Prometheus without using the OpenTelemetry Collector.

Go Example

Here's how to create OpenTelemetry metrics and expose them for Prometheus scraping in Go using the current API:

package main

import (
    "context"
    "log"
    "net/http"
    "time"

    "go.opentelemetry.io/otel"
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/metric"
    sdkmetric "go.opentelemetry.io/otel/sdk/metric"
)

func main() {
    ctx := context.Background()

    // Create Prometheus exporter
    exporter, err := prometheus.New()
    if err != nil {
        log.Fatal(err)
    }

    // Create MeterProvider with the exporter
    provider := sdkmetric.NewMeterProvider(
        sdkmetric.WithReader(exporter),
    )
    otel.SetMeterProvider(provider)

    // Get a meter
    meter := provider.Meter("example.com/metrics")

    // Create a counter
    counter, err := meter.Int64Counter(
        "requests_total",
        metric.WithDescription("Total number of requests"),
        metric.WithUnit("1"),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Create a histogram
    histogram, err := meter.Float64Histogram(
        "request_duration_seconds",
        metric.WithDescription("Request duration in seconds"),
        metric.WithUnit("s"),
    )
    if err != nil {
        log.Fatal(err)
    }

    // Simulate recording metrics
    go func() {
        for {
            counter.Add(ctx, 1, metric.WithAttributes(
                attribute.String("method", "GET"),
                attribute.String("path", "/api/users"),
            ))

            histogram.Record(ctx, 0.25, metric.WithAttributes(
                attribute.String("method", "GET"),
            ))

            time.Sleep(1 * time.Second)
        }
    }()

    // Expose /metrics endpoint with OpenTelemetry exporter
    http.Handle("/metrics", exporter)
    log.Println("Serving metrics on :8080/metrics")
    log.Fatal(http.ListenAndServe(":8080", nil))
}

Configure Prometheus to scrape the endpoint:

yaml

scrape_configs:
  - job_name: 'go-app'
    scrape_interval: 10s
    static_configs:
      - targets: ['localhost:8080']

Key changes from older API:

No need for controller or processor packages (removed in current API)
Use prometheus.New() directly without config
Use sdkmetric.NewMeterProvider() with WithReader(exporter)
Simpler metric creation with meter.Int64Counter(), meter.Float64Histogram()
Important: Use exporter directly as the HTTP handler, not promhttp.Handler() - the exporter implements http.Handler and exposes your OpenTelemetry metrics

A complete runnable example is available on GitHub.

OpenTelemetry vs Prometheus

While both OpenTelemetry and Prometheus are powerful observability tools, they have different strengths and focuses:

Feature	OpenTelemetry	Prometheus
Primary Focus	Complete observability (traces, metrics, logs)	Metrics and alerting
Data Collection	Push and pull models	Primarily pull model
Storage	Requires external storage	Includes built-in TSDB
Query Language	Depends on backend	PromQL
Instrumentation	Vendor-neutral API	Prometheus-specific

For a more detailed comparison, see OpenTelemetry vs Prometheus.

Use Cases

Migration from Prometheus to OpenTelemetry: Use the Prometheus receiver to collect existing metrics while transitioning to OpenTelemetry.
Hybrid Monitoring Systems: Leverage both Prometheus for alerting and OpenTelemetry for distributed tracing.
Extended Visualization: Export OpenTelemetry metrics to Prometheus for visualization in Grafana dashboards.
Legacy System Integration: Use Prometheus exporters for legacy systems while adopting OpenTelemetry for new applications.