Monitoring cache stats using OpenTelemetry Go Metrics

January 03, 2023

3 min read

This article explains how to use opentelemetry-go Metrics API to collect metrics, for example, go-redis/cache stats.

Getting started with OpenTelemetry Metrics

To get started with metrics, you need a MeterProvider which provides access to Meters:

import "go.opentelemetry.io/otel/metric/global"

// Meter can be a global/package variable.
var Meter = global.MeterProvider().Meter("app_or_package_name")

Using the meter, you can create instruments and use them to measure operations. The simplest Counter instrument looks like this:

import "go.opentelemetry.io/otel/metric/instrument"

counter := Meter.SyncInt64().Counter(
    "test.my_counter",
    instrument.WithUnit("1"),
    instrument.WithDescription("Just a test counter"),
)

// Increment the counter.
counter.Add(ctx, 1, attribute.String("foo", "bar"))
counter.Add(ctx, 10, attribute.String("hello", "world"))

You can find more examples at GitHub.

Cache stats

Our Redis-based cache keeps stats about hits and misses in the following struct:

type Stats struct {
    Hits   uint64
    Misses uint64
}

You can get the current stats with:

stats := cache.Stats()
fmt.Println("hits", stats.Hits)
fmt.Println("misses", stats.Misses)

Monitoring cache stats

To start monitoring our cache, we can create a separate instrument for each struct field. Here we are using CounterObserver instrument which periodically calls a function to gather stats.

import (
    "go.opentelemetry.io/otel/metric"
    "go.opentelemetry.io/otel/metric/instrument"
)

func MonitorCache(cache *cache.Cache, meter metric.Meter) {
    hits, _ := meter.AsyncInt64().Counter("cache.hits")
    misses, _ := meter.AsyncInt64().Counter("cache.misses")

    if err := meter.RegisterCallback(
        []instrument.Asynchronous{
            hits,
            misses,
        },
        // SDK periodically calls this function to collect data.
        func(ctx context.Context) {
            stats := cache.Stats()

            hits.Observe(ctx, int64(stats.Hits))
            misses.Observe(ctx, int64(stats.Misses))
        },
    ); err != nil {
        panic(err)
    }
}

Using the instruments above we get access to the following metrics:

cache.hits - number of cache hits.
cache.misses - number of cache hits.
cache.hits + cache.misses - number of cache requests.
cache.hits / (cache.hits + cache.misses) - cache hit rate.

Metric attributes

The code above works well enough, but what if we want to add another metric:

diff

type Stats struct {
    Hits   uint64
    Misses uint64
+    Errors uint64
}

We could add another instrument to observe Errors field, but then we also need to update our math:

cache.hits + cache.misses + cache.errors - number of cache requests.
cache.hits / (cache.hits + cache.misses + cache.errors) - cache hit rate.

Can we do better? Yes, using a single instrument and metric attributes:

import (
    "go.opentelemetry.io/otel/attribute"
    "go.opentelemetry.io/otel/metric"
    "go.opentelemetry.io/otel/metric/instrument"
)

func MonitorCache(cache *cache.Cache, meter metric.Meter) {
    cacheCounter, _ := meter.AsyncInt64().Counter("cache.stats")

    hits := []attribute.KeyValue{attribute.String("type", "hits")}
    misses := []attribute.KeyValue{attribute.String("type", "misses")}
    errors := []attribute.KeyValue{attribute.String("type", "errors")}

    if err := meter.RegisterCallback(
        []instrument.Asynchronous{
            cacheCounter,
        },
        // SDK periodically calls this function to collect data.
        func(ctx context.Context) {
            stats := cache.Stats()

            cacheCounter.Observe(ctx, int64(stats.Hits), hits...)
            cacheCounter.Observe(ctx, int64(stats.Misses), misses...)
            cacheCounter.Observe(ctx, int64(stats.Errors), errors...)
        },
    ); err != nil {
        panic(err)
    }
}

Our new math looks like this and does not require changes when you add new stats:

cache.stats - number of cache requests.
filter(cache.stats, type = "hits") - number of cache hits.
filter(cache.stats, type = "misses") - number of cache misses.
filter(cache.stats, type = "hits") / cache.stats - cache hit rate.

As a bonus, you can easily visualize all available metrics using grouping by type attribute, for example, using Uptrace:

text

cache.stats | group by type

Prometheus

To export metrics to Prometheuse, see Exporting OpenTelemetry Metrics to Prometheus.

What's next

Next, you can learn about the available metric instruments and try to instrument your code.

You may also be interested in: