Deploying Uptrace on Kubernetes

This comprehensive guide walks you through deploying Uptrace, an open-source APM and observability platform, on Kubernetes using Helm charts. Uptrace supports distributed tracing, metrics, and log management to help you monitor your applications effectively.

Prerequisites

Kubernetes Cluster

You'll need a running Kubernetes cluster. For local development and testing, you can create one using:

K3s - Lightweight Kubernetes distribution
Kind - Kubernetes in Docker
Minikube - Local Kubernetes cluster

Required Tools

kubectl - Kubernetes command-line tool
Helm - Kubernetes package manager

Disk Space

With the default values, the chart creates about 23Gi of persistent volumes: 9Gi for ClickHouse, 3 × 4Gi for PostgreSQL, and 2Gi for Redis. The node also needs room for the container images.

On a single-node cluster, keep the free disk space above the kubelet eviction threshold, which is 5–10% of the disk depending on the distribution. Below it, the node reports DiskPressure and evicts the Uptrace pods. See Pods are evicted or stay Pending.

The Uptrace Helm Chart

The Uptrace Helm chart includes the Uptrace application and all required dependencies:

Redis (for caching)
PostgreSQL (for metadata storage)
ClickHouse (for observability data)
OpenTelemetry Collector (for metrics collection)

Initial Setup

Add Uptrace Helm Repository

First, add the Uptrace Helm repository to your local Helm installation:

shell

helm repo add uptrace https://charts.uptrace.dev --force-update

Create Monitoring Namespace

Create a dedicated namespace for Uptrace and its dependencies. This provides logical separation and makes resource management easier:

shell

kubectl create namespace monitoring

Note: You can delete all resources created in this guide by removing the namespace:

shell

kubectl delete namespace monitoring

Configuration Management

Understanding values.yaml Files

Helm uses values.yaml files to customize chart deployments without modifying the original templates. These files act as a central configuration point for defining variables that populate Helm template placeholders.

View Default Configuration

To see all available configuration options for the Uptrace chart:

shell

helm show values uptrace/uptrace --devel

Get Sample Configuration Files

For complete configuration examples, clone the Uptrace Helm charts repository:

shell

git clone https://github.com/uptrace/helm-charts.git
cd helm-charts
cat uptrace-values.yaml

This repository contains sample *-values.yaml files for all components.

Secret Key

The chart generates a random service.secret on the first install, stores it in a Kubernetes Secret, and keeps it on later upgrades. To use your own value, set uptrace.secret. Set it also when a tool such as Argo CD renders the chart with helm template, because such a render cannot read the existing Secret. The chart refuses to render when uptrace.config.service.secret is FIXME or empty.

Dependency Setup

Redis Configuration

Uptrace uses Redis for in-memory caching. Since caching data is ephemeral, no persistent storage is required. A Redis instance with 32MB of free RAM is sufficient.

Using the Built-in Redis

The Uptrace Helm chart includes a built-in Redis server. Enable it in your uptrace-values.yaml:

yaml

redis:
  enabled: true

The chart will automatically configure Uptrace to connect to this Redis instance. You can customize it further:

yaml

redis:
  enabled: true
  password: '' # optional password
  persistence:
    enabled: true
    size: 2Gi

Connecting to Redis

To connect to your Redis instance for testing:

shell

kubectl port-forward service/uptrace-redis 6379:6379 -n monitoring
redis-cli

Using Existing Redis

If you have an existing Redis database, keep the built-in Redis disabled and configure Uptrace to use your instance:

yaml

redis:
  enabled: false

uptrace:
  config:
    redis_cache:
      addrs:
        alpha: 'your-redis-host:6379'

PostgreSQL Configuration

Uptrace uses PostgreSQL to store metadata including users, projects, and monitors. The metadata volume is typically small. The chart creates a PostgreSQL cluster of 3 instances with a 4Gi volume each. For local testing, one instance is enough:

yaml

postgresql:
  instances: 1

Installing PostgreSQL

Install PostgreSQL using the CloudNativePG operator, which provides enterprise-grade PostgreSQL management:

shell

# Add the CloudNativePG repository
helm repo add cnpg https://cloudnative-pg.github.io/charts

# Install the PostgreSQL operator
helm upgrade --install cnpg \
  --namespace cnpg-system \
  --create-namespace \
  cnpg/cloudnative-pg

Verify the installation:

shell

kubectl get all -n cnpg-system

Using Existing PostgreSQL

If you have an existing PostgreSQL database, disable the bundled one and configure your connection details:

yaml

postgresql:
  enabled: false

uptrace:
  config:
    pg:
      addr: 'your-postgresql-host:5432'
      user: uptrace
      password: your-password
      database: uptrace

ClickHouse Configuration

ClickHouse stores all observability data including spans, logs, events, and metrics. Start with a pod having 4 CPUs, 1GB RAM, and 10GB disk space, then scale vertically as needed.

Installing ClickHouse

Install ClickHouse using the Altinity operator:

shell

kubectl apply -f https://raw.githubusercontent.com/Altinity/clickhouse-operator/master/deploy/operator/clickhouse-operator-install-bundle.yaml

Verify the operator is running:

shell

kubectl get pods -n kube-system | grep clickhouse-operator

Using Existing ClickHouse

For existing ClickHouse installations, disable the bundled one and configure your cluster details:

yaml

clickhouse:
  enabled: false

uptrace:
  config:
    ch_cluster:
      cluster: 'your-cluster-name'
      replicated: false
      distributed: false

      shards:
        - replicas:
            - addr: 'your-clickhouse-host:9000'
              database: uptrace
              user: uptrace
              password: your-password
              dial_timeout: 3s
              write_timeout: 5s
              max_retries: 3
              max_execution_time: 15s
              query_settings:
                session_timezone: UTC
                async_insert: 1
                query_cache_nondeterministic_function_handling: 'save'
                allow_suspicious_types_in_group_by: 1
                allow_suspicious_types_in_order_by: 1

Troubleshooting: If ClickHouse encounters issues in Kubernetes, refer to this troubleshooting guide.

OpenTelemetry Collector Setup

The chart uses the OpenTelemetry Operator to deploy collectors that gather pod metrics and other observability data.

Install cert-manager

OpenTelemetry Operator requires cert-manager for certificate management:

shell

helm repo add jetstack https://charts.jetstack.io --force-update
helm install cert-manager jetstack/cert-manager \
  --namespace cert-manager \
  --create-namespace \
  --version v1.17.2 \
  --set crds.enabled=true

Install OpenTelemetry Operator

shell

helm repo add open-telemetry https://open-telemetry.github.io/opentelemetry-helm-charts --force-update
helm install otel-operator open-telemetry/opentelemetry-operator \
  --set "manager.collectorImage.repository=otel/opentelemetry-collector-k8s" \
  --set "manager.collectorImage.tag=0.154.0" \
  --set admissionWebhooks.certManager.enabled=false \
  --set admissionWebhooks.autoGenerateCert.enabled=true \
  --namespace opentelemetry \
  --create-namespace

Disable OpenTelemetry Collector (Optional)

If you prefer to manage OpenTelemetry Collector separately, disable it in your uptrace-values.yaml:

yaml

otelcol:
  enabled: false

otelcolDaemonset:
  enabled: false

Installing Uptrace

With all dependencies configured, install Uptrace using your custom values file:

shell

helm install uptrace uptrace/uptrace \
  -f uptrace-values.yaml \
  -n monitoring \
  --devel

Verify Installation

A first install takes several minutes, mostly because CloudNativePG initializes the PostgreSQL instances one after another.

A post-install Job runs uptrace migrate up, and Helm deletes the Job when it succeeds. Until it completes, the Uptrace pods exit with a message that names migrate up, and Kubernetes restarts them, so a few restarts on a first install are expected.

If ClickHouse restarts after Uptrace has started, Uptrace logs ClickHouse replica is still down every few seconds. Uptrace keeps checking the replica and logs ClickHouse replica is back up when it recovers, which can take a few minutes.

Check that all resources are running:

shell

kubectl get all -n monitoring

View Uptrace application logs:

shell

kubectl logs uptrace-0 -n monitoring

When the install completes, the namespace contains these pods:

text

chi-uptrace1-uptrace1-0-0-0         2/2     Running
otelcol-74fc9966fb-k5fvc            1/1     Running
otelcol-daemonset-collector-vq9xw   1/1     Running
uptrace-0                           1/1     Running
uptrace-postgresql-1                1/1     Running
uptrace-postgresql-2                1/1     Running
uptrace-postgresql-3                1/1     Running
uptrace-redis-0                     1/1     Running

Database Connections

Connect to ClickHouse for troubleshooting:

shell

kubectl port-forward service/chi-uptrace1-uptrace1-0-0 9000:9000 -n monitoring
clickhouse-client

Accessing Uptrace

Local Access Setup

Uptrace will be available at http://uptrace.local with these default credentials:

Username: admin@uptrace.local
Password: admin

Add the domain to your /etc/hosts file:

text

127.0.0.1 uptrace.local

Port Forwarding

To open Uptrace without an ingress controller or a hosts entry, forward a local port to the Uptrace service:

shell

kubectl port-forward service/uptrace 8080:80 -n monitoring

Then open http://localhost:8080.

Platform-Specific Configuration

K3s Setup

K3s ships with the Traefik ingress controller, and the chart's default ingress works with it. Point uptrace.local at the node in your /etc/hosts file:

text

127.0.0.1 uptrace.local  # Replace with the node IP if K3s runs on another host

Minikube Setup

Enable the ingress controller:

shell

minikube addons enable ingress

Wait for ingress pods to be ready:

shell

kubectl get pods -n ingress-nginx

Get Minikube's IP address:

shell

minikube ip

Update your /etc/hosts file with the Minikube IP:

text

192.168.49.2 uptrace.local  # Replace with your minikube ip

AWS EKS Deployment

For external access on AWS EKS using the AWS Load Balancer Controller, add these annotations to your uptrace-values.yaml:

yaml

service:
  type: LoadBalancer
  port: 80
  loadBalancerSourceRanges:
    - '0.0.0.0/0' # Restrict this to your IP ranges for security
  annotations:
    service.beta.kubernetes.io/aws-load-balancer-type: 'external'
    service.beta.kubernetes.io/aws-load-balancer-nlb-target-type: 'ip'
    service.beta.kubernetes.io/aws-load-balancer-target-group-attributes: 'preserve_client_ip.enabled=true'
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: 'http'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-protocol: 'http'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-port: '80'
    service.beta.kubernetes.io/aws-load-balancer-healthcheck-path: '/'

Scaling and Maintenance

Horizontal Scaling

Scale Uptrace by increasing replicas in your uptrace-values.yaml:

yaml

uptrace:
  replicaCount: 3 # Increase based on your load requirements

Apply the changes:

shell

helm upgrade uptrace uptrace/uptrace \
  -f uptrace-values.yaml \
  -n monitoring \
  --devel

Upgrading Uptrace

Check the supported upgrade paths first. If you run v2.0 or a v2.1 beta, also read upgrading from v2.0 or a v2.1 beta.

Check Current Version

Before upgrading, verify the installed version and check what's available:

shell

# Current chart and app version
helm list -n monitoring

# Latest available version
helm repo update uptrace
helm search repo uptrace/uptrace --devel

Check the latest release notes on GitHub Releases.

Back Up Databases

Always create backups of both PostgreSQL and ClickHouse before upgrading.

PostgreSQL:

For a logical backup that works with the default chart, run pg_dump from a temporary PostgreSQL client pod. The command reads connection details from the CloudNativePG application secret created by the chart:

shell

kubectl run uptrace-pg-dump -n monitoring --rm -i --restart=Never \
  --image=postgres:17 \
  --env="PGHOST=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.host}' | base64 -d)" \
  --env="PGPORT=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.port}' | base64 -d)" \
  --env="PGUSER=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.username}' | base64 -d)" \
  --env="PGPASSWORD=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.password}' | base64 -d)" \
  --env="PGDATABASE=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.dbname}' | base64 -d)" \
  --command -- pg_dump > uptrace-pg-backup-$(date +%Y%m%d).sql

If you have configured CloudNativePG backups, you can also trigger an on-demand backup. The default chart creates the PostgreSQL cluster as uptrace-postgresql:

shell

kubectl apply -f - <<EOF
apiVersion: postgresql.cnpg.io/v1
kind: Backup
metadata:
  name: uptrace-pg-pre-upgrade
  namespace: monitoring
spec:
  cluster:
    name: uptrace-postgresql
EOF

ClickHouse:

The default chart runs the stock ClickHouse server image and does not install clickhouse-backup. Back up ClickHouse with your Kubernetes storage backup process, such as CSI VolumeSnapshot, or install and manage a ClickHouse backup tool as part of your deployment.

If you manage clickhouse-backup yourself and it is available in the ClickHouse pod:

shell

kubectl exec -n monitoring chi-uptrace1-uptrace1-0-0-0 -- \
  clickhouse-backup create uptrace-backup-$(date +%Y%m%d)

Run the Upgrade

Update the Helm repository and upgrade the release:

shell

helm repo update uptrace
helm upgrade uptrace uptrace/uptrace \
  -f uptrace-values.yaml \
  -n monitoring \
  --wait \
  --devel

Helm automatically validates the config, runs database migrations, and restarts Uptrace. The migrations run in two Jobs around the rollout. Before it, a pre-upgrade Job runs uptrace migrate up --before-release-change with the new image, while the old pods keep serving. After it, a post-upgrade Job runs uptrace migrate up --wait 10m. The --wait flag of helm upgrade makes Helm start the post-upgrade Job only after the new pods are ready. To check the migration status, recover from a failed migration, or roll back, see upgrading Uptrace.

To install a specific chart version, use the --version flag:

shell

helm search repo uptrace/uptrace --devel --versions

helm upgrade uptrace uptrace/uptrace \
  -f uptrace-values.yaml \
  -n monitoring \
  --version <chart-version> \
  --wait \
  --devel

Verify the Upgrade

After the upgrade completes, confirm everything is running:

shell

# Check pod status and restarts
kubectl get pods -n monitoring

# View the Uptrace application logs for migration output
kubectl logs uptrace-0 -n monitoring --since=5m

# Verify the health endpoint
kubectl exec -n monitoring uptrace-0 -- wget -qO- http://localhost:80/health/ready

Rolling Back

If the pre-upgrade Job fails, Helm stops the upgrade before the rollout, and the old pods keep serving. Fix the cause and run helm upgrade again.

After the pre-upgrade Job completed, the migrations changed the databases. Roll them back with the down steps of the new image, as rolling back on Kubernetes describes:

Run uptrace migrate down --before-release-change as a Job with the new image.

Change the image back without the hooks of the old revision, and wait for the pods:

shell

helm history uptrace -n monitoring
helm rollback uptrace <revision> -n monitoring --no-hooks --wait

Run uptrace migrate down as a Job with the new image.

Do not run helm rollback with hooks: the old revision's Jobs use the old image, which does not contain the down steps of the new migrations.

If the down steps fail, or the data matters more than the time to restore it, scale Uptrace down, remove the existing application-owned PostgreSQL objects, restore from your backup, and then roll back:

shell

# Stop Uptrace so it can't write during the restore
kubectl scale statefulset uptrace -n monitoring --replicas=0

# Recreate PostgreSQL from the backup
kubectl run uptrace-pg-restore -n monitoring --rm -i --restart=Never \
  --image=postgres:17 \
  --env="PGHOST=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.host}' | base64 -d)" \
  --env="PGPORT=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.port}' | base64 -d)" \
  --env="PGUSER=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.username}' | base64 -d)" \
  --env="PGPASSWORD=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.password}' | base64 -d)" \
  --env="PGDATABASE=$(kubectl get secret -n monitoring uptrace-postgresql-app -o jsonpath='{.data.dbname}' | base64 -d)" \
  --command -- sh -c \
  'psql -v ON_ERROR_STOP=1 --dbname="$PGDATABASE" -c "DROP OWNED BY CURRENT_USER CASCADE" && psql -v ON_ERROR_STOP=1 --dbname="$PGDATABASE"' \
  < uptrace-pg-backup-YYYYMMDD.sql

# Roll back to the previous release after PostgreSQL is restored
helm rollback uptrace -n monitoring --no-hooks

Restore ClickHouse using the same backup method you used before the upgrade. If you manage clickhouse-backup yourself:

shell

kubectl exec -n monitoring chi-uptrace1-uptrace1-0-0-0 -- \
  clickhouse-backup restore uptrace-backup-YYYYMMDD

Resource Management

Monitor Resource Usage

Keep an eye on resource consumption:

shell

kubectl top pods -n monitoring
kubectl top nodes

Adjust Resource Limits

Configure resource requests and limits in your uptrace-values.yaml:

yaml

uptrace:
  resources:
    requests:
      memory: '512Mi'
      cpu: '500m'
    limits:
      memory: '1Gi'
      cpu: '1000m'

Uptrace sets the soft memory limit of the Go runtime to service.memory_limit_ratio (default 0.9) × the container memory limit, and caps the default buffer of each pipeline at a quarter of that. Set a memory limit so that this applies.

When a pod stops, /health/ready fails at once. To let the load balancer stop routing to the pod before its ports close, set service.shutdown.delay to cover the drain time of your load balancer or ingress:

yaml

uptrace:
  config:
    service:
      shutdown:
        delay: 10s

The delay adds to the shutdown stages (servers_timeout, drain_timeout, and close_timeout, 30s together by default). Keep the sum below the pod's terminationGracePeriodSeconds (30s by default), or Kubernetes kills the process before it finishes.

Troubleshooting

Common Issues

Pods not starting: Check resource availability and node capacity, including disk space
Database connection errors: Verify database credentials and network policies
Ingress not working: Ensure ingress controller is properly installed and configured
Performance issues: Monitor resource usage and scale components as needed

Pods Are Evicted or Stay Pending

When the node runs low on disk space, the kubelet sets the DiskPressure condition, evicts the pods, and does not schedule new ones. helm install --wait then fails with context deadline exceeded, and the events show messages such as The node was low on resource: ephemeral-storage or The node had condition: [DiskPressure].

Check the node condition:

shell

kubectl describe node | grep DiskPressure

To recover:

Free disk space on the node, for example by removing unused container images.
Wait until DiskPressure is False. The kubelet clears the condition a few minutes after the space is freed.
Remove the failed release and reinstall Uptrace. To start from empty databases, also delete the persistent volumes as described in Uninstall Uptrace.

Useful Commands

View all resources in the monitoring namespace:

shell

kubectl get all -n monitoring -o wide

Describe problematic pods:

shell

kubectl describe pod <pod-name> -n monitoring

Check events for troubleshooting:

shell

kubectl get events -n monitoring --sort-by='.lastTimestamp'

Cleanup

Uninstall Uptrace

Remove the Uptrace release:

shell

helm uninstall uptrace -n monitoring

The ClickHouse operator removes the ClickHouse installation after Helm returns, so it can stay in the Terminating state for a minute. Wait for it before you install Uptrace again:

shell

kubectl wait --for=delete chi/uptrace1 -n monitoring --timeout=5m

helm uninstall keeps the data, so a later install starts with the same databases and secret:

the persistent volume claims of ClickHouse, PostgreSQL, and Redis;
the uptrace-secret Secret with the generated secret key.

To start from scratch, delete them too:

shell

kubectl delete pvc --all -n monitoring
kubectl delete secret uptrace-secret -n monitoring

Complete Cleanup

Delete the entire monitoring namespace and all resources:

shell

kubectl delete namespace monitoring

Remove Operators (Optional)

If you installed operators specifically for this deployment:

shell

# Remove OpenTelemetry Operator
helm uninstall otel-operator -n opentelemetry
kubectl delete namespace opentelemetry

# Remove cert-manager
helm uninstall cert-manager -n cert-manager
kubectl delete namespace cert-manager

# Remove CloudNativePG Operator
helm uninstall cnpg -n cnpg-system
kubectl delete namespace cnpg-system

Alternative Deployment Methods

Kubernetes is one of several deployment options for Uptrace:

Docker - Quick deployment for development and small-scale production
DEB/RPM packages - Traditional server deployments
Ansible - Automated bare metal deployments

Choose the method that best fits your infrastructure and requirements.

Next Steps

Once Uptrace is running successfully:

Configure Applications: Set up your applications to send telemetry data to Uptrace
Create Dashboards: Build custom dashboards for your specific monitoring needs
Set Up Alerts: Configure alerting rules for critical metrics and traces
Explore Features: Discover advanced features like distributed tracing visualization and log correlation
Performance Tuning: Optimize configuration based on your data volume and query patterns