Uptrace Configuration Reference

Uptrace is an open-source APM and observability platform that supports distributed tracing, metrics, and log management. This guide covers comprehensive configuration options for customizing Uptrace settings and optimizing the ClickHouse database schema through a single YAML configuration file.

Configuration File Management

Creating and Locating Configuration Files

Generate a default configuration file using the Uptrace binary:

shell

./uptrace config create -config=/path/to/config.yml

Specify the configuration file location using either method:

CLI argument (recommended):

shell

uptrace --config=/etc/uptrace/config.yml serve

Environment variable:

shell

export UPTRACE_CONFIG=/etc/uptrace/config.yml
uptrace serve

If no configuration path is specified, Uptrace defaults to /etc/uptrace/config.yml.

Environment Variable Integration

Enhance security and flexibility by using environment variables in your YAML configuration:

yml

pg:
  addr: ${UPTRACE_PG_ADDR:localhost:5432}
  user: ${UPTRACE_PG_USER:uptrace}
  password: ${UPTRACE_PG_PASSWORD}
  database: ${UPTRACE_PG_DATABASE:uptrace}

Environment Variable Syntax:

Simple substitution: ${ENV_VAR_NAME}
With fallback defaults: ${ENV_VAR_NAME:default_value}
Escape dollar signs: Use $$ to literal $

Variables are expanded using Go's os.Expand function before YAML parsing.

Database Architecture and Configuration

Uptrace uses a dual-database architecture optimized for different data types and access patterns.

PostgreSQL: Metadata Storage

PostgreSQL stores application metadata including user accounts, project settings, dashboards, and alert configurations. This database typically requires only a few megabytes of storage.

Basic Configuration:

yml

pg:
  addr: localhost:5432
  user: uptrace
  password: uptrace
  database: uptrace

  # TLS configuration for secure connections
  # Recommended for production deployments
  #tls:
  #  insecure_skip_verify: true  # WARNING: Only for self-signed certificates

High Availability with Read Replicas:

yml

pg:
  addr: primary-host:5432
  user: uptrace
  password: uptrace
  database: uptrace

  # Standby replicas
  replicas:
    - standby-host1:5432
    - standby-host2:5432

Using connection string:

yaml

pg:
  dsn: postgresql://user:password@host:5432/database?sslmode=verify-full

ClickHouse: Telemetry Data Storage

ClickHouse serves as the primary storage engine for high-volume telemetry data including spans, logs, and metrics. It's optimized for analytical queries and time-series data.

Basic Configuration:

yml

ch_cluster:
  # Cluster name for ClickHouse operations
  cluster: 'uptrace1'

  # High availability features
  replicated: false    # Enable for automatic failover
  distributed: false   # Enable for horizontal scaling (Premium only)

  shards:
    - replicas:
        - addr: localhost:9000
          user: default
          password: ''
          database: uptrace

          # Connection and query timeouts
          dial_timeout: 3s
          write_timeout: 5s
          max_retries: 3
          max_execution_time: 15s

Production High Availability Setup:

yml

ch_cluster:
  cluster: 'uptrace1'
  replicated: true

  shards:
    - replicas:
        - addr: clickhouse-1:9000
          user: default
          password: ''
          database: uptrace

        - addr: clickhouse-2:9000
          user: default
          password: ''
          database: uptrace

        - addr: clickhouse-3:9000
          user: default
          password: ''
          database: uptrace

Using connection string:

yaml

ch_cluster:
  shards:
    - replicas:
        - dsn: clickhouse://your_username:your_password@your_host:9000/your_database

          # Connection and query timeouts
          dial_timeout: 3s
          write_timeout: 5s
          max_retries: 3
          max_execution_time: 15s

ClickHouse Advanced Configuration

Query Performance Tuning:

yml

ch_cluster:
  shards:
    - replicas:
        - addr: localhost:9000
          # ClickHouse-specific query settings
          query_settings:
            async_insert: 1               # Enable async inserts (default: enabled)
            wait_for_async_insert: 1      # Wait for insert completion (default: disabled)
            max_memory_usage: 10000000000 # 10GB memory limit per query

TLS Configuration for ClickHouse:

First, configure ClickHouse server to enable secure TCP port:

xml

<!-- ClickHouse server configuration -->
<?xml version="1.0" ?>
<clickhouse>
  <tcp_port_secure>9440</tcp_port_secure>

  <openSSL>
    <server>
      <certificateFile>/etc/clickhouse/certs/chnode.crt</certificateFile>
      <privateKeyFile>/etc/clickhouse/certs/chnode.key</privateKeyFile>
      <verificationMode>relaxed</verificationMode>
      <caConfig>/etc/clickhouse/certs/Uptrace_CA.crt</caConfig>
      <cacheSessions>true</cacheSessions>
      <disableProtocols>sslv2,sslv3</disableProtocols>
      <preferServerCiphers>true</preferServerCiphers>
    </server>
  </openSSL>
</clickhouse>

Then configure Uptrace to use TLS:

yml

ch_cluster:
  shards:
    - replicas:
        - addr: localhost:9440  # Use secure port
          user: default
          database: uptrace

          tls:
            ca_file: /etc/clickhouse/certs/Uptrace_CA.crt
            cert_file: /etc/clickhouse/certs/chnode.crt
            key_file: /etc/clickhouse/certs/chnode.key
            insecure_skip_verify: false

ClickHouse Cloud Configuration:

yml

ch_cluster:
  shards:
    - replicas:
        - addr: tm849a32za.us-central1.gcp.clickhouse.cloud:9440
          user: default
          password: your_cloud_password
          database: uptrace
          tls:
            # Override server name if address doesn't match certificate
            server_name_override: 'tm849a32za.us-central1.gcp.clickhouse.cloud'

Schema Configuration and Optimization

Customize ClickHouse schema settings for optimal performance and storage efficiency.

⚠️ Critical: Schema changes require database reset and will delete all existing data:

shell

uptrace ch reset

Storage and Compression Settings

yml

ch_schema:
  # Compression algorithm selection
  # LZ4: Fast compression/decompression, moderate ratio
  # ZSTD(1): Better compression ratio, slightly slower
  # Default: ClickHouse default compression
  compression: ZSTD(1)

  # Storage policies for different data types
  # Configure different storage tiers (SSD, HDD, S3) based on access patterns
  spans_index: { storage_policy: hot_storage }    # Frequently accessed span indexes
  spans_data: { storage_policy: warm_storage }    # Span detail data
  span_links: { storage_policy: cold_storage }    # Span relationship data
  logs_index: { storage_policy: hot_storage }     # Log search indexes
  logs_data: { storage_policy: warm_storage }     # Log detail data
  events_index: { storage_policy: hot_storage }   # Event search indexes
  events_data: { storage_policy: warm_storage }   # Event detail data
  metrics: { storage_policy: hot_storage }        # Metrics time-series data

ClickHouse Replication Setup

Step 1: Configure ClickHouse Cluster

Create a cluster configuration with at least 3 replicas. Critical: Set internal_replication = true to prevent data duplication.

xml

<clickhouse>
  <remote_servers>
    <uptrace1>
      <shard>
        <internal_replication>true</internal_replication>
        <replica>
          <host>clickhouse-1</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>clickhouse-2</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>clickhouse-3</host>
          <port>9000</port>
        </replica>
      </shard>
    </uptrace1>
  </remote_servers>
</clickhouse>

Step 2: Enable Replication in Uptrace

yml

ch_cluster:
  cluster: 'uptrace1'
  replicated: true  # Enable ClickHouse table replication

  shards:
    - replicas:
        - addr: clickhouse-1:9000
        - addr: clickhouse-2:9000
        - addr: clickhouse-3:9000

Step 3: Apply Configuration Changes

shell

uptrace ch reset

Step 4: Verify Replication Status

sql

SELECT
    database,
    table,
    is_leader,
    replica_is_active
FROM system.replicas;

Expected output showing active replicas:

text

┌─database─┬─table───────────┬─is_leader─┬─replica_is_active────────────────────────┐
│ uptrace  │ spans_data      │         1 │ {'replica1':1,'replica2':1,'replica3':1} │
│ uptrace  │ spans_index     │         1 │ {'replica1':1,'replica2':1,'replica3':1} │
└──────────┴─────────────────┴───────────┴──────────────────────────────────────────┘

Network and Domain Configuration

Port Configuration

Configure network listeners for different protocols:

yml

listen:
  # HTTP server: OTLP/HTTP API, REST API, and web interface
  http:
    addr: ':80'  # Standard HTTP port

  # gRPC server: OTLP/gRPC for high-performance telemetry ingestion
  grpc:
    addr: ':4317'  # Standard OTLP gRPC port

  # TLS configuration for both HTTP and gRPC
  #tls:
  #  cert_file: /etc/uptrace/uptrace.crt
  #  key_file: /etc/uptrace/uptrace.key

Domain and URL Configuration

Primary Site Configuration:

yml

site:
  # Primary URL for user interface and API access
  # Used for: dashboard links, CORS validation, authentication redirects
  # Must be accessible from all clients and match reverse proxy setup
  url: https://uptrace.mycompany.com

Separate Ingestion Endpoint:

yml

site:
  url: https://uptrace.mycompany.com

  # Dedicated URL for telemetry data ingestion
  # Benefits: load balancing, security policies, CDN optimization
  # Defaults to site.url if not specified
  ingest_url: https://ingest.mycompany.com

Security and TLS Configuration

Automatic TLS with Let's Encrypt

For production deployments with automatic certificate management:

yml

##
## Automatic TLS certificate management via Let's Encrypt
##
certmagic:
  enabled: true                 # Enable automatic certificate issuance
  staging_ca: false            # Use production Let's Encrypt (set true for testing)
  http_challenge_addr: ':80'   # Port for HTTP-01 challenge validation

Manual TLS Certificate Configuration

Generate Self-Signed Certificate (Development):

shell

openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout uptrace.key -out uptrace.crt -subj "/CN=localhost" \
  -addext "subjectAltName=DNS:localhost,DNS:uptrace.local"

Add to System Trust Store:

shell

sudo cp uptrace.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

Configure Uptrace with Custom Certificates:

yml

listen:
  grpc:
    addr: ':4317'
  http:
    addr: ':443'
  tls:
    cert_file: /etc/uptrace/tls/uptrace.crt
    key_file: /etc/uptrace/tls/uptrace.key
    min_version: '1.2'  # Minimum TLS version
    #ca_file: /etc/uptrace/tls/ca.crt  # Custom CA if needed

site:
  url: 'https://uptrace.mycompany.com'

Database TLS Configuration

Complete TLS Client Options:

yml

# PostgreSQL TLS example
pg:
  addr: localhost:5432
  user: uptrace
  password: uptrace
  database: uptrace
  tls:
    # Certificate files (optional for client authentication)
    cert_file: /etc/uptrace/client.crt
    key_file: /etc/uptrace/client.key
    ca_file: /etc/uptrace/ca.crt

    # TLS version constraints
    min_version: '1.2'
    max_version: '1.3'

    # Security settings
    insecure_skip_verify: false  # Verify server certificates
    server_name_override: ''     # Override server name verification

Common TLS Configurations:

yml

# Use system certificate store
pg:
  tls: {}

# Skip certificate verification (insecure, testing only)
pg:
  tls:
    insecure_skip_verify: true

# Disable TLS entirely
pg:
  tls:
    insecure: true

User Management and Authentication

Initial Setup Configuration

Customize default users and organizational structure:

yml

##
## Bootstrap data created on first startup
## Modify before first run - changes afterward require manual management
##
seed_data:
  users:
    - key: admin_user
      name: System Administrator
      email: admin@mycompany.com
      password: ChangeThisPassword123!  # Use strong password
      email_confirmed: true

  user_tokens:
    - key: admin_token
      user: admin_user
      token: secure_admin_token_here    # Generate random token

  orgs:
    - key: main_org
      name: My Company

  org_users:
    - key: admin_org_membership
      org: main_org
      user: admin_user
      role: owner                       # Roles: owner, admin, member

  projects:
    - key: production_project
      name: Production Environment
      org: main_org

  project_tokens:
    - key: prod_token
      project: production_project
      token: prod_telemetry_secret      # Used in OTLP DSN

  project_users:
    - key: admin_project_access
      project: production_project
      user: admin_user
      perm_level: admin                 # Levels: admin, editor, viewer

Authentication Controls

yml

auth:
  # Disable username/password authentication (SSO-only mode)
  # SSO providers remain available when disabled
  disabled: false

  # Restrict registration by email domain
  # Examples:
  # - Single domain: '^.+@mycompany\.com$'
  # - Multiple domains: '^.+@(company1|company2)\.com$'
  # - Subdomain wildcard: '^.+@.*\.mycompany\.com$'
  email_regexp: '^.+@mycompany\.com$'

Single Sign-On Integration

Uptrace supports enterprise identity providers:

Okta - Enterprise SSO platform
Keycloak - Open-source identity management
Cloudflare Access - Zero Trust security
Google Workspace - Google enterprise accounts

Data Processing and Performance Tuning

Telemetry Data Processing Configuration

Optimize processing for different telemetry data types based on your ingestion volume:

yml

##
## Spans (distributed tracing data)
##
spans:
  # Processing parallelism (default: number of CPU cores)
  # Increase for high-volume tracing workloads
  max_threads: 16

  # Database insertion batch size
  # Larger batches improve throughput but increase memory usage
  max_insert_size: 20000

  # In-memory buffer capacity
  # Spans are dropped when buffer is full - monitor for drops
  max_buffered_records: 200000

##
## Logs (application log data)
##
logs:
  max_threads: 12
  max_insert_size: 15000
  max_buffered_records: 150000

##
## Metrics (time-series data)
##
metrics:
  max_threads: 8
  max_insert_size: 10000
  max_buffered_records: 100000

  # Cumulative to delta conversion capacity
  # Affects OpenTelemetry cumulative counter processing
  max_cumulative_timeseries: 2000000

##
## Events (custom application events)
##
events:
  max_threads: 4
  max_insert_size: 5000
  max_buffered_records: 50000

##
## Span Links (cross-trace relationships)
##
span_links:
  # Disable if not using span links to save resources
  disabled: false
  max_threads: 2
  max_insert_size: 5000
  max_buffered_records: 25000

Query Performance Limits

Prevent resource exhaustion from expensive queries:

yml

##
## Trace query resource limits
##
trace:
  # Maximum spans returned per query
  # Prevents UI timeouts and excessive memory usage
  query_limit: 500000

  # Maximum memory per query (bytes)
  # Prevents out-of-memory errors
  max_memory_usage_bytes: 500000000  # 500MB

Caching Configuration

Redis caching improves query performance and reduces database load:

yml

##
## Redis cache for query performance optimization
##
redis_cache:
  # Redis server addresses (cluster support)
  addrs:
    1: redis-1:6379
    2: redis-2:6379

  # Authentication
  username: ''
  password: 'redis_password'
  db: 0

  # Connection settings
  dial_timeout: 5s
  read_timeout: 3s
  write_timeout: 3s

  # TLS for secure connections
  #tls:
  #  insecure_skip_verify: false

Feature Management

Optional Feature Modules

Enable or disable features based on your needs:

yml

##
## Monitoring and alerting system
##
alerting:
  # Disable to save resources if not using alerts
  disabled: false

##
## Service dependency graph generation
##
service_graph:
  # Disable to save CPU/memory if not using service maps
  disabled: false

##
## JavaScript sourcemap processing
##
sourcemaps:
  # Disable in air-gapped environments or if not tracking JS errors
  # Requires internet access to download sourcemaps
  disabled: false

##
## Internal telemetry collection
##
monitoring:
  # Disable self-monitoring to prevent internal telemetry data
  disabled: false
  # DSN for sending Uptrace's own telemetry
  dsn: http://project1_secret@uptrace.local/2?grpc=4317

Notification and Communication

Email Configuration

Configure SMTP for alert notifications and user management:

yml

##
## SMTP mailer configuration
## Required for: alerts, password resets, user invitations
##
mailer:
  smtp:
    enabled: true
    host: smtp.mycompany.com
    port: 587                    # Common ports: 25, 465, 587
    username: uptrace@mycompany.com
    password: smtp_password
    from: 'Uptrace Alerts <uptrace@mycompany.com>'

    # TLS settings
    #tls:
    #  disabled: false           # Enable TLS (recommended)

Gmail Configuration Example:

Generate app-specific password in Gmail settings
Configure Uptrace:

yml

mailer:
  smtp:
    enabled: true
    host: smtp.gmail.com
    port: 587
    username: youraccount@gmail.com
    password: your_app_specific_password  # Not your regular password
    from: 'youraccount@gmail.com'

Telegram Integration

Enable Telegram notifications for alerts:

yml

##
## Telegram bot configuration
## Setup guide: https://core.telegram.org/bots#6-botfather
##
telegram:
  # Bot token from @BotFather
  bot_token: 'your_telegram_bot_token'

System Configuration

Application Logging

Control Uptrace's internal logging:

yml

##
## Application logging configuration
##
logging:
  # Log levels: DEBUG, INFO, WARN, ERROR
  # DEBUG: Verbose output for troubleshooting
  # INFO: Standard operational information
  # WARN: Warnings and errors only
  # ERROR: Errors only
  level: INFO

Premium Features

Enable advanced features with a license:

yml

##
## Premium Edition license
## Enables: distributed tables, advanced SSO, priority support
##
license:
  key: 'your_premium_license_key'

Premium Features Include:

ClickHouse distributed tables for horizontal scaling
Advanced SSO integrations
Priority technical support
Enhanced security features

Deployment Patterns

Production Checklist

Security Requirements:

✅ Change all default passwords and tokens in seed_data
✅ Configure TLS/SSL certificates for all connections
✅ Set up authentication restrictions (auth.email_regexp)
✅ Enable secure database connections
✅ Configure proper firewall rules

High Availability Setup:

✅ Deploy multiple Uptrace instances behind load balancer
✅ Configure PostgreSQL with read replicas
✅ Set up ClickHouse replication (minimum 3 nodes)
✅ Configure Redis for caching and session storage
✅ Set up database backup procedures

Performance Optimization:

✅ Tune telemetry processing settings based on volume
✅ Configure appropriate resource limits
✅ Set up monitoring for Uptrace itself
✅ Configure data retention policies
✅ Optimize ClickHouse storage policies

Operational Requirements:

✅ Configure email notifications for alerts
✅ Set up log aggregation for Uptrace logs
✅ Document recovery procedures
✅ Configure backup verification
✅ Set up monitoring dashboards

Reverse Proxy Configuration

When deploying behind a reverse proxy, ensure proper URL configuration:

yml

site:
  # Main deployment
  url: https://uptrace.mycompany.com

  # Subpath deployment
  url: https://mycompany.com/uptrace

Load Balancer Health Checks:

HTTP endpoint: GET /api/health
Expected response: 200 OK

Scaling Guidelines

Horizontal Scaling:

Uptrace instances are stateless and can be scaled horizontally
Use load balancer to distribute traffic across instances
Each instance should connect to the same PostgreSQL and ClickHouse databases

Vertical Scaling Recommendations: According to ClickHouse documentation, prefer vertical scaling over horizontal scaling:

Successful deployments with ClickHouse use servers with hundreds of cores, terabytes of RAM, and petabytes of disk space. Scaling vertically first provides cost efficiency, lower operational overhead, and better query performance.

Resource Requirements:

Uptrace instances: 2-4 CPU cores, 4-8GB RAM per instance
PostgreSQL: 2-4 CPU cores, 4-8GB RAM, SSD storage
ClickHouse: Scale based on data volume (prefer large instances)
Redis: 1-2 CPU cores, 2-4GB RAM

Troubleshooting Guide

Common Configuration Issues

Database Connection Problems:

yml

# Check connection settings
pg:
  addr: localhost:5432  # Verify host and port
  user: uptrace         # Ensure user exists
  password: uptrace     # Verify password
  database: uptrace     # Ensure database exists

TLS Certificate Issues:

bash

# Verify certificate validity
openssl x509 -in uptrace.crt -text -noout

# Check certificate expiration
openssl x509 -in uptrace.crt -checkend 86400

# Test TLS connection
openssl s_client -connect uptrace.local:443

Performance Problems:

Monitor ClickHouse resource usage
Check for dropped telemetry data in metrics
Review query performance in ClickHouse logs
Verify network latency between components

Authentication Issues:

Verify email regex patterns if configured
Check SSO provider configuration
Confirm user permissions and project access
Review authentication logs for error details

Monitoring Uptrace Health

Key Metrics to Monitor:

Database connection health
Telemetry data ingestion rates
Query response times
Memory and CPU usage
Disk space utilization

Health Check Endpoints:

API health: GET /api/health
Database connectivity: Monitor connection metrics
Redis connectivity: Monitor cache hit rates

K8s deploymentThis comprehensive guide walks you through deploying Uptrace, an open-source APM and observability platform, on Kubernetes using Helm charts. Uptrace supports distributed tracing, metrics, and log management to help you monitor your applications effectively.

Troubleshooting