OpenTelemetry Collector Prometheus [full guide with code]

You can use OpenTelemetry Collector to collect Prometheus metrics using Prometheus receiver and export OpenTelemetry metrics to Prometheus using Promethes exporter.

What is OpenTelemetry?

OpenTelemetry is an open source and vendor-neutral API for distributed tracing (including logs and errors) and OpenTelemetry metrics. With OpenTelemetry, you can instrument your application once and then add or change OpenTelemetry-compatible backends without changing the instrumentation.

What is OpenTelemetry Collector?

OpenTelemetry Collector is a proxy between your application and a backend that receives telemetry data, transforms it, and exports the data to various observability platforms, storage systems, or other destinations.

Collector can also act as an agent that pulls telemetry data from monitored systems, for example, Redis or host metrics.

OpenTelemetry and Prometheus

OpenTelemetry and Prometheus have different data formats, but they can be integrated to work together.

You can configure OpenTelemetry Collector to accept OpenTelemetry data and then export it to Prometheus using the Prometheus remote write protocol.

You can also do the opposite and use OpenTelemetry Collector to pull Prometheus metrics and the export them to OpenTelemetry backends using the OpenTelemetry Protocol.

Prometheus receiver

You can use OpenTelemetry Collector to pull Prometheus metrics using Prometheus receiver:

When collecting metrics from Prometheus, OpenTelemetry Collector acts as a Prometheus server itself, scraping metrics from one or more Prometheus exporters, and then processing and exporting the collected metrics to configured destinations, for example, Uptrace APM.

yaml
receivers:
  prometheus_simple:
    collection_interval: 10s
    endpoint: '172.17.0.5:9153'
    metrics_path: '/metrics'
    use_service_account: false
    tls:
      ca_file: '/path/to/ca'
      cert_file: '/path/to/cert'
      key_file: '/path/to/key'
      insecure_skip_verify: true

exporters:
  otlp/uptrace:
    endpoint: otlp.uptrace.dev:4317
    headers:
      # Copy your project DSN here
      uptrace-dsn: '<FIXME>'

service:
  pipelines:
    metrics:
      receivers: [prometheus_simple]
      exporters: [otlp/uptrace]

Prometheus exporter

OpenTelemetry Collector Prometheus Exporter is a component that allows users to export metrics from the OpenTelemetry Collector to the Prometheus monitoring system.

To use the OpenTelemetry Collector Prometheus Exporter, you need to configure it with the desired configuration parameters.

The exporter allows you to add custom labels and metadata to the exported metrics, which can be useful for organizing and categorizing metrics in Prometheus.

yaml
exporters:
  prometheus:
    endpoint: '1.2.3.4:1234'
    namespace: test-space
    const_labels:
      label1: value1
      'another label': spaced value
    send_timestamps: true
    metric_expiration: 180m
    resource_to_telemetry_conversion:
      enabled: true

service:
  pipelines:
    metrics:
      receivers: [otlp]
      exporters: [prometheus]

Once the exporter is configured and enabled, it will collect metrics from the OpenTelemetry Collector and forward them to the specified Prometheus server. Users can then use Prometheus tools to query, visualize, and alert on the collected metrics.

Sending metrics from Go to Prometheus

OpenTelemetry specifies how to collect, aggregate, and send metrics to backend platforms. Using OpenTelemetry instruments), you can create counter, gauge, and histogram metrics.

The simplest Counter instrument looks like this:

go
import "go.opentelemetry.io/otel/metric/instrument"

counter := Meter.SyncInt64().Counter(
    "test.my_counter",
    instrument.WithUnit("1"),
    instrument.WithDescription("Just a test counter"),
)

// Increment the counter.
counter.Add(ctx, 1, attribute.String("foo", "bar"))
counter.Add(ctx, 10, attribute.String("hello", "world"))

To send OpenTelemetry Go metrics to Prometheus, you need to provide a Prometheus target:

go
import (
    "go.opentelemetry.io/otel/exporters/prometheus"
    "go.opentelemetry.io/otel/metric/global"
    "go.opentelemetry.io/otel/sdk/export/metric/aggregation"
    "go.opentelemetry.io/otel/sdk/metric/aggregator/histogram"
    controller "go.opentelemetry.io/otel/sdk/metric/controller/basic"
    processor "go.opentelemetry.io/otel/sdk/metric/processor/basic"
    selector "go.opentelemetry.io/otel/sdk/metric/selector/simple"
)

func configureOpentelemetry() {
    exporter := configureMetrics()

    http.HandleFunc("/metrics", exporter.ServeHTTP)
    fmt.Println("listenening on http://localhost:8088/metrics")

    go func() {
        _ = http.ListenAndServe(":8088", nil)
    }()
}

func configureMetrics() *prometheus.Exporter {
    config := prometheus.Config{}

    ctrl := controller.New(
        processor.NewFactory(
            selector.NewWithHistogramDistribution(
                histogram.WithExplicitBoundaries(config.DefaultHistogramBoundaries),
            ),
            export.CumulativeExportKindSelector(),
            processor.WithMemory(true),
        ),
    )

    exporter, err := prometheus.New(config, ctrl)
    if err != nil {
        panic(err)
    }

    global.SetMeterProvider(exporter.MeterProvider())

    return exporter
}

And then configure Prometheus to scrape exposed metrics:

yaml
scrape_configs:
  - job_name: 'opentelemetry'
    static_configs:
      - targets: ['localhost:8088']

You can find a runnable example at GitHub.

Uptrace

Uptrace is an open source APM with an intuitive query builder, rich dashboards, alerting rules, and integrations for most languages and frameworks. It can process billions of spans and metrics on a single server and allows to monitor your applications at 10x lower cost.

Uptrace uses ClickHouse database to store traces, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.

You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.

See also