OpenTelemetry Metrics [with examples]

OpenTelemetry Metrics is a standard on how to collect, aggregate, and send metrics to OpenTelemetry APM tools such as Uptrace or Prometheus.

While defining a new standard, OpenTelemetry also aims to work with existing metrics instrumentation protocols such as Prometheus and Statsd. Furthermore, OpenTelemetry Collector supports even more protocols like AWS Metrics, InfluxDB, Chrony, etc.

OpenTelemetry also allows you to correlate metrics and traces via exemplars which should show you a broader picture of the state of your system.

What are metrics?

Metrics are numerical data points that represent the health and performance of your system, such as CPU utilization, network traffic, and database connections.

You can use metrics to measure, monitor, and compare performance, for example, you can measure server response time, memory utilization, error rate, and more.

Instruments

An instrument is a specific type of metric (e.g., counter, gauge, histogram) that you use to collect data about a particular aspect of your application's behavior.

You capture measurements by creating instruments that have:

• An unique name, for example, http.server.duration.
• An instrument kind, for example, Histogram.
• An optional unit of measure, for example, milliseconds or bytes.
• An optional description.

Timeseries

A single instrument can produce multiple timeseries. A timeseries is a metric with an unique set of attributes, for example, each host has a separate timeseries for the same metric name.

Additive instruments

Additive or summable instruments produce timeseries that, when added up together, produce another meaningful and accurate timeseries. Additive instruments that measure non-decreasing numbers are also called monotonic.

For example, http.server.requests is an additive timeseries, because you can sum the number of requests from different hosts to get the total number of requests.

But system.memory.utilization (percents) is not additive, because the sum of memory utilization from different hosts is not meaningful (90% + 90% = 180%).

Synchronous instruments

Synchronous instruments are invoked together with operations they are measuring. For example, to measure the number of requests, you can call counter.Add(ctx, 1) whenever there is a new request. Synchronous measurements can have an associated trace context.

For synchronous instruments the difference between additive and grouping instruments is that additive instruments produce summable timeseries and grouping instruments produce a histogram.

Asynchronous instruments

Asynchronous instruments (observers) periodically invoke a callback function to collect measurements. For example, you can use observers to periodically measure memory or CPU usage. Asynchronous measurements can't have an associated trace context.

When choosing between UpDownCounterObserver (additive) and GaugeObserver (grouping), choose UpDownCounterObserver for summable timeseries and GaugeObserver otherwise. For example, to measure system.memory.usage (bytes), you should use UpDownCounterObserver. But to measure system.memory.utilization (percents), you should use GaugeObserver.

Choosing instruments

1. If you need a histogram, a heatmap, or percentiles, use Histogram.
2. If you want to count something by recording a delta value:
   • If the value is monotonic, use Counter.
   • Otherwise, use UpDownCounter.
3. If you want to measure something by recording an absolute value:
   • If the value is additive/summable:
     • If the value is monotonic, use CounterObserver.
     • Otherwise, use UpDownCounterObserver.
   • If the value is NOT additive/summable, use GaugeObserver

Counter

import "go.opentelemetry.io/otel/metric"

emailCounter, _ := meter.Int64Counter(
    "some.prefix.emails",
    metric.WithDescription("Number of sent emails"),
)

emailCounter.Add(ctx, 1)

import "go.opentelemetry.io/otel/metric"

opHistogram, _ := meter.Int64Histogram(
    "some.prefix.duration",
    metric.WithDescription("Duration of some operation"),
)

t1 := time.Now()
op(ctx)
dur := time.Since(t1)

opHistogram.Record(ctx, dur.Microseconds())

import "go.opentelemetry.io/otel/metric"

counter, _ := meter.Int64ObservableCounter("some.prefix.cache")

// Arbitrary key/value labels.
hits := []attribute.KeyValue{attribute.String("type", "hits")}
misses := []attribute.KeyValue{attribute.String("type", "misses")}
errors := []attribute.KeyValue{attribute.String("type", "errors")}

if _, err := meter.RegisterCallback(
    func(ctx context.Context, o metric.Observer) error {
        stats := cache.Stats()

        o.ObserveInt64(counter, stats.Hits, metric.WithAttributes(hits...))
        o.ObserveInt64(counter, stats.Misses, metric.WithAttributes(misses...))
        o.ObserveInt64(counter, stats.Errors, metric.WithAttributes(errors...))

        return nil
    },
    counter,
); err != nil {
    panic(err)
}

import "go.opentelemetry.io/otel/metric"

errorRate, _ := meter.Float64ObservableGauge("some.prefix.error_rate")

if _, err := meter.RegisterCallback(
    func(ctx context.Context, o metric.Observer) error {
        o.ObserveFloat64(errorRate, rand.Float64())
        return nil
    },
    errorRate,
); err != nil {
    panic(err)
}