OpenTelemetry Operator

OpenTelemetry Operator is a Kubernetes-native application that automates the deployment, management, and configuration of OpenTelemetry components within a Kubernetes cluster. It extends the Kubernetes API to include custom resources for OpenTelemetry, allowing you to define and manage your observability stack using familiar Kubernetes concepts.

OpenTelemetry Operator

How OpenTelemetry Operator works?

OpenTelemetry Operator serves as an automation layer between your Kubernetes cluster and OpenTelemetry components, simplifying the deployment and management of observability infrastructure.

The main functions of OpenTelemetry Operator include:

  1. Automating the deployment of OpenTelemetry Collectors
  2. Managing the lifecycle of OpenTelemetry components
  3. Providing auto-instrumentation capabilities for applications
  4. Simplifying configuration management for OpenTelemetry in Kubernetes

OpenTelemetry Operator works by introducing new custom resource definitions (CRDs) to your Kubernetes cluster and watching for changes to these resources.

When you create or modify OpenTelemetry-related custom resources, the operator takes action to ensure the desired state is achieved, managing the deployment and configuration of OpenTelemetry Collectors and instrumentations based on your specifications.

When to use OpenTelemetry Operator?

While manually deploying OpenTelemetry components in Kubernetes is possible, using the OpenTelemetry Operator offers several advantages:

  • Automated deployment and configuration of OpenTelemetry Collectors
  • Simplified management of OpenTelemetry components lifecycle
  • Easy scaling of observability infrastructure
  • Automatic instrumentation of applications
  • Centralized configuration management
  • Kubernetes-native approach to observability

Installation

To install OpenTelemetry Operator in your Kubernetes cluster, you can use Helm or kubectl. Here's how to install it:

Using Helm

Helm is often the preferred method for installing the OpenTelemetry Operator. Here's how to do it:

# Add the OpenTelemetry Helm repository
helm repo add open-telemetry https://github.com/open-telemetry/opentelemetry-helm-charts

# Update your Helm repositories
helm repo update

# Install the OpenTelemetry Operator
helm install my-otel-operator open-telemetry/opentelemetry-operator

Using kubectl

Alternatively, you can install the OpenTelemetry Operator using kubectl:

# Apply the OpenTelemetry Operator manifests
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml

After installation, you can verify the operator's status using:

kubectl get pods -n opentelemetry-operator-system

This should show the operator pod in a Running state.

For troubleshooting and monitoring your Kubernetes resources, including the OpenTelemetry Operator, you might find these guide helpful:

Configuration

OpenTelemetry Operator uses Custom Resource Definitions (CRDs) to configure OpenTelemetry components in your Kubernetes cluster. Here's an example of how to deploy an OpenTelemetry Collector using the operator:

TIP

Don't forget to add the Uptrace exporter to the spec.config.exporters section and include it in the appropriate pipeline(s) in the spec.config.service.pipelines section.

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: my-collector
spec:
  config: |
    receivers:
      otlp:
        protocols:
          grpc:
          http:

    processors:
      batch:

    exporters:
      otlp/uptrace:
        endpoint: otlp.uptrace.dev:4317
        headers:
          uptrace-dsn: 'https://<token>@uptrace.dev/<project_id>'

    service:
      pipelines:
        traces:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp/uptrace]
        metrics:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp/uptrace]
        logs:
          receivers: [otlp]
          processors: [batch]
          exporters: [otlp/uptrace]

Troubleshooting

If the OpenTelemetry Operator is not working as expected, you can check the operator logs for potential issues:

kubectl logs -n opentelemetry-operator-system -l app.kubernetes.io/name=opentelemetry-operator

You can also check the status of the OpenTelemetryCollector resources:

kubectl get opentelemetrycollectors

Auto-instrumentation

One of the powerful features of OpenTelemetry Operator is its ability to automatically instrument applications in your Kubernetes cluster. Here's an example of how to set up auto-instrumentation:

apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
  name: my-instrumentation
spec:
  exporter:
    endpoint: http://my-collector:4317
  propagators:
    - tracecontext
    - baggage
  sampler:
    type: parentbased_traceidratio
    argument: '1'

Best Practices and Use Cases

Scaling OpenTelemetry with Kubernetes

OpenTelemetry Operator makes it easy to scale your observability infrastructure. You can adjust the number of collector replicas:

apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
  name: my-collector
spec:
  replicas: 3
  # ... rest of the configuration

Multi-cluster observability

For multi-cluster setups, you can deploy the OpenTelemetry Operator in each cluster and configure collectors to forward data to a central Uptrace instance:

exporters:
  otlp:
    endpoint: central-collector.example.com:4317
  otlp/uptrace:
    endpoint: otlp.uptrace.dev:4317
    headers:
      uptrace-dsn: 'https://<token>@uptrace.dev/<project_id>'
service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [batch]
      exporters: [otlp, otlp/uptrace]

Advanced Configuration Options

Custom Resource Definitions (CRDs)

OpenTelemetry Operator introduces several CRDs:

  • OpenTelemetryCollector: Defines collector instances
  • Instrumentation: Configures auto-instrumentation
  • OpenTelemetryCollectorWebhook: Allows for custom admission webhooks

You can explore these CRDs using:

kubectl get crd | grep opentelemetry.io

Security considerations

To enhance the security of your OpenTelemetry setup:

  1. Use RBAC to limit access to OpenTelemetry resources:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: opentelemetry-viewer
rules:
  - apiGroups: ['opentelemetry.io']
    resources: ['opentelemetrycollectors', 'instrumentations']
    verbs: ['get', 'list', 'watch']
  1. Enable TLS for collector communications:
receivers:
  otlp:
    protocols:
      grpc:
        tls:
          cert_file: /path/to/cert
          key_file: /path/to/key
  1. Use Kubernetes secrets for sensitive information like Uptrace DSN.

Uptrace

Uptrace is an OpenTelemetry APMopen in new window that helps developers pinpoint failures and find performance bottlenecks. Uptrace can process billions of spans on a single server and allows to monitor your software at 10x lower cost.

You can get startedopen in new window with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.

Last Updated: