Exporting OpenTelemetry Metrics to Prometheus

What is Prometheus?

Prometheus is an open-source monitoring system with flexible query language and real-time alerting. Prometheus records metrics in a pluggable time series database using a HTTP pull model.

What is OpenTelemetry?

OpenTelemetry is an open source and vendor-neutral API for distributed tracing (including logs and errors) and metrics. With OpenTelemetry, you can instrument your application once and then add or change OpenTelemetry-compatible backends without changing the instrumentation.

OpenTelemetry Collector

What is OpenTelemetry Collector?

OpenTelemetry Collector is a proxy between your application and a backend that receives telemetry data, transforms it, and then exports to backends that can store the data permanently.

Collector can also act as an agent that pulls telemetry data from monitored systems, for example, Redis or filesystem metrics.

Collecting Prometheus metrics

You can use OpenTelemetry Collector to pull Prometheus metrics using Prometheus receiveropen in new window:

receivers:
  prometheus_simple:
    collection_interval: 10s
    endpoint: '172.17.0.5:9153'
    metrics_path: '/metrics'
    use_service_account: false
    tls:
      ca_file: '/path/to/ca'
      cert_file: '/path/to/cert'
      key_file: '/path/to/key'
      insecure_skip_verify: true

exporters:
  otlp:
    endpoint: otlp.uptrace.dev:4317
    headers:
      # Copy your project DSN here
      uptrace-dsn: 'https://<token>@uptrace.dev/<project_id>'

service:
  pipelines:
    metrics:
      receivers: [prometheus_simple]
      exporters: [otlp]

Exporting OpenTelemetry metrics to Prometheus

You can also configure OpenTelemetry Collector to export data to Prometheus using Prometheus exporteropen in new window:

exporters:
  prometheus:
    endpoint: '1.2.3.4:1234'
    namespace: test-space
    const_labels:
      label1: value1
      'another label': spaced value
    send_timestamps: true
    metric_expiration: 180m
    resource_to_telemetry_conversion:
      enabled: true

OpenTelemetry Metrics

OpenTelemetry Go metrics

OpenTelemetry specifies how to collect, aggregate, and send metrics to backend platforms. Using OpenTelemetry instruments, you can create counter, gauge, and histogram metrics.

The simplest Counter instrument looks like this:

import "go.opentelemetry.io/otel/metric/instrument"

counter := Meter.SyncInt64().Counter(
	"test.my_counter",
    instrument.WithUnit("1"),
	instrument.WithDescription("Just a test counter"),
)

// Increment the counter.
counter.Add(ctx, 1, attribute.String("foo", "bar"))
counter.Add(ctx, 10, attribute.String("hello", "world"))

Sending Go metrics to Prometheus

To send OpenTelemetry Go metric to Prometheus, you need to provide a Prometheus target:

import (
	"go.opentelemetry.io/otel/exporters/prometheus"
	"go.opentelemetry.io/otel/metric/global"
	"go.opentelemetry.io/otel/sdk/export/metric/aggregation"
	"go.opentelemetry.io/otel/sdk/metric/aggregator/histogram"
	controller "go.opentelemetry.io/otel/sdk/metric/controller/basic"
	processor "go.opentelemetry.io/otel/sdk/metric/processor/basic"
	selector "go.opentelemetry.io/otel/sdk/metric/selector/simple"
)

func configureOpentelemetry() {
	exporter := configureMetrics()

	http.HandleFunc("/metrics", exporter.ServeHTTP)
	fmt.Println("listenening on http://localhost:8088/metrics")

	go func() {
		_ = http.ListenAndServe(":8088", nil)
	}()
}

func configureMetrics() *prometheus.Exporter {
	config := prometheus.Config{}

	ctrl := controller.New(
		processor.NewFactory(
			selector.NewWithHistogramDistribution(
				histogram.WithExplicitBoundaries(config.DefaultHistogramBoundaries),
			),
			export.CumulativeExportKindSelector(),
			processor.WithMemory(true),
		),
	)

	exporter, err := prometheus.New(config, ctrl)
	if err != nil {
		panic(err)
	}

	global.SetMeterProvider(exporter.MeterProvider())

	return exporter
}

And then configure Prometheus to scrape exposed metrics:

scrape_configs:
  - job_name: 'opentelemetry'
    static_configs:
      - targets: ['localhost:8088']

You can find a runnable example at GitHubopen in new window.

Storing data

Uptrace is an open source DataDog competitoropen in new window with an intuitive query builder, rich dashboards, alerting rules, and integrations for most languages and frameworks. It can process billions of spans and metrics on a single server and allows to monitor your applications at 10x lower cost.

Uptrace uses ClickHouse database to store traces, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.

You can get startedopen in new window with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.

See also

Last Updated: