OpenTelemetry Operator
OpenTelemetry Operator is a Kubernetes-native application that automates the deployment, management, and configuration of OpenTelemetry components within a Kubernetes cluster. It extends the Kubernetes API to include custom resources for OpenTelemetry, allowing you to define and manage your observability stack using familiar Kubernetes concepts.
How OpenTelemetry Operator works?
OpenTelemetry Operator serves as an automation layer between your Kubernetes cluster and OpenTelemetry components, simplifying the deployment and management of observability infrastructure.
The main functions of OpenTelemetry Operator include:
- Automating the deployment of OpenTelemetry Collectors
- Managing the lifecycle of OpenTelemetry components
- Providing auto-instrumentation capabilities for applications
- Simplifying configuration management for OpenTelemetry in Kubernetes
OpenTelemetry Operator works by introducing new custom resource definitions (CRDs) to your Kubernetes cluster and watching for changes to these resources.
When you create or modify OpenTelemetry-related custom resources, the operator takes action to ensure the desired state is achieved, managing the deployment and configuration of OpenTelemetry Collectors and instrumentations based on your specifications.
When to use OpenTelemetry Operator?
While manually deploying OpenTelemetry components in Kubernetes is possible, using the OpenTelemetry Operator offers several advantages:
- Automated deployment and configuration of OpenTelemetry Collectors
- Simplified management of OpenTelemetry components lifecycle
- Easy scaling of observability infrastructure
- Automatic instrumentation of applications
- Centralized configuration management
- Kubernetes-native approach to observability
Installation
To install OpenTelemetry Operator in your Kubernetes cluster, you can use Helm or kubectl. Here's how to install it:
Using Helm
Helm is often the preferred method for installing the OpenTelemetry Operator. Here's how to do it:
# Add the OpenTelemetry Helm repository
helm repo add open-telemetry https://github.com/open-telemetry/opentelemetry-helm-charts
# Update your Helm repositories
helm repo update
# Install the OpenTelemetry Operator
helm install my-otel-operator open-telemetry/opentelemetry-operator
Using kubectl
Alternatively, you can install the OpenTelemetry Operator using kubectl:
# Apply the OpenTelemetry Operator manifests
kubectl apply -f https://github.com/open-telemetry/opentelemetry-operator/releases/latest/download/opentelemetry-operator.yaml
After installation, you can verify the operator's status using:
kubectl get pods -n opentelemetry-operator-system
This should show the operator pod in a Running state.
For troubleshooting and monitoring your Kubernetes resources, including the OpenTelemetry Operator, you might find these guide helpful:
Configuration
OpenTelemetry Operator uses Custom Resource Definitions (CRDs) to configure OpenTelemetry components in your Kubernetes cluster. Here's an example of how to deploy an OpenTelemetry Collector using the operator:
TIP
Don't forget to add the Uptrace exporter to the spec.config.exporters section and include it in the appropriate pipeline(s) in the spec.config.service.pipelines section.
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: my-collector
spec:
config: |
receivers:
otlp:
protocols:
grpc:
http:
processors:
batch:
exporters:
otlp/uptrace:
endpoint: otlp.uptrace.dev:4317
headers:
uptrace-dsn: 'https://<token>@uptrace.dev/<project_id>'
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/uptrace]
metrics:
receivers: [otlp]
processors: [batch]
exporters: [otlp/uptrace]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp/uptrace]
Troubleshooting
If the OpenTelemetry Operator is not working as expected, you can check the operator logs for potential issues:
kubectl logs -n opentelemetry-operator-system -l app.kubernetes.io/name=opentelemetry-operator
You can also check the status of the OpenTelemetryCollector resources:
kubectl get opentelemetrycollectors
Auto-instrumentation
One of the powerful features of OpenTelemetry Operator is its ability to automatically instrument applications in your Kubernetes cluster. Here's an example of how to set up auto-instrumentation:
apiVersion: opentelemetry.io/v1alpha1
kind: Instrumentation
metadata:
name: my-instrumentation
spec:
exporter:
endpoint: http://my-collector:4317
propagators:
- tracecontext
- baggage
sampler:
type: parentbased_traceidratio
argument: '1'
Best Practices and Use Cases
Scaling OpenTelemetry with Kubernetes
OpenTelemetry Operator makes it easy to scale your observability infrastructure. You can adjust the number of collector replicas:
apiVersion: opentelemetry.io/v1alpha1
kind: OpenTelemetryCollector
metadata:
name: my-collector
spec:
replicas: 3
# ... rest of the configuration
Multi-cluster observability
For multi-cluster setups, you can deploy the OpenTelemetry Operator in each cluster and configure collectors to forward data to a central Uptrace instance:
exporters:
otlp:
endpoint: central-collector.example.com:4317
otlp/uptrace:
endpoint: otlp.uptrace.dev:4317
headers:
uptrace-dsn: 'https://<token>@uptrace.dev/<project_id>'
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp, otlp/uptrace]
Advanced Configuration Options
Custom Resource Definitions (CRDs)
OpenTelemetry Operator introduces several CRDs:
- OpenTelemetryCollector: Defines collector instances
- Instrumentation: Configures auto-instrumentation
- OpenTelemetryCollectorWebhook: Allows for custom admission webhooks
You can explore these CRDs using:
kubectl get crd | grep opentelemetry.io
Security considerations
To enhance the security of your OpenTelemetry setup:
- Use RBAC to limit access to OpenTelemetry resources:
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: opentelemetry-viewer
rules:
- apiGroups: ['opentelemetry.io']
resources: ['opentelemetrycollectors', 'instrumentations']
verbs: ['get', 'list', 'watch']
- Enable TLS for collector communications:
receivers:
otlp:
protocols:
grpc:
tls:
cert_file: /path/to/cert
key_file: /path/to/key
- Use Kubernetes secrets for sensitive information like Uptrace DSN.
Uptrace
Uptrace is an OpenTelemetry APM that helps developers pinpoint failures and find performance bottlenecks. Uptrace can process billions of spans on a single server and allows to monitor your software at 10x lower cost.
You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.
