Monitoring Relay Redis client with OpenTelemetry

Relay is a Redis client for PHP that can be used as a drop-in replacement for PhpRedis and Predis, except two orders of magnitude faster.

This article explains how to use OpenTelemetry and Uptrace to monitor Relay Redis client.

What is Relay?

Relayopen in new window is a PHP extension that is both a Redis client and a shared in-memory cache. Think APCu and PhpRedis having a child — but with magic powers.

It uses server-assisted client side caching to actively invalidate its in-memory cache and informs you about invalidations, so your app can update its runtime cache mid-request.

<?php

$relay = new Relay(host: '127.0.0.1', port: 6379);

// Fetch the user count from Relay’s memory,
// or from Redis if the key has not been cached, yet.
$users = $relay->get('users:count');

// Listen to all invalidation events.
$relay->onInvalidated(function (Relay\Event $event) use ($users) {
    if ($event->key === 'users:count') {
        $users = null;
    }
});

What is tracing?

Distributed Tracingopen in new window allows you to see how a request progresses through different services and components, timings of every operation, any logs and errors as they occur. In a distributed environment, tracing also helps you understand relationships and interactions between distributed micro-services and systems.

Spans and trace

Using tracing, you can break down requests into spansopen in new window. Span is an operation (unit of work) your app performs handling a request, for example, a database query or a network call.

Trace is a tree of spans that shows the path that a request makes through an app. Root span is the first span in a trace.

Spans and trace

To learn more about tracing, see Distributed tracing using OpenTelemetryopen in new window.

What is OpenTelemetry?

OpenTelemetryopen in new window is an open-source observability framework for distributed tracingopen in new window (including logs and errors) and OpenTelemetry metricsopen in new window.

Otel allows developers to collect and export telemetry data in a vendor agnostic way. With OpenTelemetry, you can instrument your application once and then add or change vendors without changing the instrumentation, for example, here is a list popular DataDog alternativesopen in new window that support OpenTelemetry.

OpenTelemetry is available for most programming languages and provides interoperability across different languages and environments.

OpenTelemetry API

OpenTelemetry API is a programming interface that you can use to instrument code and collect telemetry data such as traces, metrics, and logs.

To measure performance of database queries or HTTP requests, you can create a span using OpenTelemetry PHP API:

$tracerProvider = $uptrace->createTracerProvider();
$tracer = $tracerProvider->getTracer('app_or_package_name');

function insertUser(string $name, string $email) {
    $span = $tracer->spanBuilder('insert-user')->startSpan();
    $spanScope = $span->activate();

    DB::insert('insert into users (id, name) values (?, ?)', [$name, $email]);

    $spanScope->detach();
    $span->end();
}

To record contextual information, you can annotate spans with attributesopen in new window:

function insertUser(string $name, string $email) {
    $span = $tracer->spanBuilder('insert-user')->startSpan();
    $spanScope = $span->activate();

    // Check if the span is recording and set attributes.
    if ($span.isRecording()) {
        $span->setAttribute('enduser.name', $name);
        $span->setAttribute('enduser.email', $email);
    }

    DB::insert('insert into users (name, email) values (?, ?)', [$name, $email]);

    $spanScope->detach();
    $span->end();
}

See OpenTelemetry PHP tracingopen in new window for details.

What is Uptrace?

Uptrace is an open-source APM toolopen in new window that supports distributed tracing, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.

Uptrace uses OpenTelelemetry to collect data and ClickHouse database to store it. ClickHouse is the only dependency.

Relay OpenTelemetry instrumentation

Instrumentations are plugins for popular frameworks and libraries that use OpenTelemetry API to record important operations, for example, HTTP requests, DB queries, logs, errors, and more.

Relay provides the RelayOpenTelemetryopen in new window instrumentation to instrument Redis commands:

use \CacheWerk\Relay\Psr\Tracing\RelayOpenTelemetry;

$redis = new RelayOpenTelemetry(function() {
    return new Relay\Relay;
}, $tracerProvider);

The $tracerProvider can be created by Uptrace like this:

use \CacheWerk\Relay\Psr\Tracing\RelayOpenTelemetry;

$uptrace = new Uptrace\Distro(...);
$tracerProvider = $uptrace->createTracerProvider();

$redis = new RelayOpenTelemetry(function() {
    return new Relay\Relay;
}, $tracerProvider);

To see Uptrace in action, you can run this GitHub exampleopen in new window:

Relay trace

Alerting and notifications

You can also use Uptrace to monitor incoming spans and send notifications via email, Slack, Telegram, and more.

For example, the following rule creates an alert when Uptrace receives any spans from Redis (where system = 'db:redis'):

alerting:
  rules:
    - name: Redis is working
      metrics:
        - uptrace.tracing.spans as $spans
      query:
        - $spans > 0
        - where system = 'db:redis'
      for: 5m
      annotations:
        summary: 'Got {{ $values.spans }} spans from Redis'

Or you can create an alert only for failed operations (status='error'):

alerting:
  rules:
    - name: Redis is failing
      metrics:
        - uptrace.tracing.spans as $spans
      query:
        - $spans{status='error'} > 10
        - where system = 'db:redis'
      for: 5m

See Converting spans to metricsopen in new window for details.

Redis monitoring

You can also monitor Redis performanceopen in new window metrics By installing OpenTelemetry Collector.

OpenTelemetry Collectoropen in new window is an agent that pulls telemetry data from systems you want to monitor and sends it to APM tools using the OpenTelemetry protocol (OTLP).

When telemetry data reaches Uptrace, it automatically generates a Redis dashboard from a pre-defined template.

Redis dashboard