Uptrace Configuration Reference

Uptrace is an open-source APM and observability platform that supports distributed tracing, metrics, and log management. This guide covers comprehensive configuration options for customizing Uptrace settings and optimizing the ClickHouse database schema through a single YAML configuration file.

Configuration File Management

Creating and Locating Configuration Files

Generate a default configuration file using the Uptrace binary:

shell
./uptrace config create -config=/path/to/config.yml

Specify the configuration file location using either method:

CLI argument (recommended):

shell
uptrace --config=/etc/uptrace/config.yml serve

Environment variable:

shell
export UPTRACE_CONFIG=/etc/uptrace/config.yml
uptrace serve

If no configuration path is specified, Uptrace defaults to /etc/uptrace/config.yml.

Environment Variable Integration

Enhance security and flexibility by using environment variables in your YAML configuration:

yml
pg:
  addr: ${UPTRACE_PG_ADDR:localhost:5432}
  user: ${UPTRACE_PG_USER:uptrace}
  password: ${UPTRACE_PG_PASSWORD}
  database: ${UPTRACE_PG_DATABASE:uptrace}

Environment Variable Syntax:

  • Simple substitution: ${ENV_VAR_NAME}
  • With fallback defaults: ${ENV_VAR_NAME:default_value}
  • Escape dollar signs: Use $$ to literal $

Variables are expanded using Go's os.Expand function before YAML parsing.

Database Architecture and Configuration

Uptrace uses a dual-database architecture optimized for different data types and access patterns.

PostgreSQL: Metadata Storage

PostgreSQL stores application metadata including user accounts, project settings, dashboards, and alert configurations. This database typically requires only a few megabytes of storage.

Basic Configuration:

yml
pg:
  addr: localhost:5432
  user: uptrace
  password: uptrace
  database: uptrace

  # TLS configuration for secure connections
  # Recommended for production deployments
  #tls:
  #  insecure_skip_verify: true  # WARNING: Only for self-signed certificates

High Availability with Read Replicas:

yml
pg:
  addr: primary-host:5432
  user: uptrace
  password: uptrace
  database: uptrace

  # Standby replicas
  replicas:
    - standby-host1:5432
    - standby-host2:5432

Using connection string:

yaml
pg:
  dsn: postgresql://user:password@host:5432/database?sslmode=verify-full

ClickHouse: Telemetry Data Storage

ClickHouse serves as the primary storage engine for high-volume telemetry data including spans, logs, and metrics. It's optimized for analytical queries and time-series data.

Basic Configuration:

yml
ch_cluster:
  # Cluster name for ClickHouse operations
  cluster: 'uptrace1'

  # High availability features
  replicated: false    # Enable for automatic failover
  distributed: false   # Enable for horizontal scaling (Premium only)

  shards:
    - replicas:
        - addr: localhost:9000
          user: default
          password: ''
          database: uptrace

          # Connection and query timeouts
          dial_timeout: 3s
          write_timeout: 5s
          max_retries: 3
          max_execution_time: 15s

Production High Availability Setup:

yml
ch_cluster:
  cluster: 'uptrace1'
  replicated: true

  shards:
    - replicas:
        - addr: clickhouse-1:9000
          user: default
          password: ''
          database: uptrace

        - addr: clickhouse-2:9000
          user: default
          password: ''
          database: uptrace

        - addr: clickhouse-3:9000
          user: default
          password: ''
          database: uptrace

Using connection string:

yaml
ch_cluster:
  shards:
    - replicas:
        - dsn: clickhouse://your_username:your_password@your_host:9000/your_database

          # Connection and query timeouts
          dial_timeout: 3s
          write_timeout: 5s
          max_retries: 3
          max_execution_time: 15s

ClickHouse Advanced Configuration

Query Performance Tuning:

yml
ch_cluster:
  shards:
    - replicas:
        - addr: localhost:9000
          # ClickHouse-specific query settings
          query_settings:
            async_insert: 1               # Enable async inserts (default: enabled)
            wait_for_async_insert: 1      # Wait for insert completion (default: disabled)
            max_memory_usage: 10000000000 # 10GB memory limit per query

TLS Configuration for ClickHouse:

First, configure ClickHouse server to enable secure TCP port:

xml
<!-- ClickHouse server configuration -->
<?xml version="1.0" ?>
<clickhouse>
  <tcp_port_secure>9440</tcp_port_secure>

  <openSSL>
    <server>
      <certificateFile>/etc/clickhouse/certs/chnode.crt</certificateFile>
      <privateKeyFile>/etc/clickhouse/certs/chnode.key</privateKeyFile>
      <verificationMode>relaxed</verificationMode>
      <caConfig>/etc/clickhouse/certs/Uptrace_CA.crt</caConfig>
      <cacheSessions>true</cacheSessions>
      <disableProtocols>sslv2,sslv3</disableProtocols>
      <preferServerCiphers>true</preferServerCiphers>
    </server>
  </openSSL>
</clickhouse>

Then configure Uptrace to use TLS:

yml
ch_cluster:
  shards:
    - replicas:
        - addr: localhost:9440  # Use secure port
          user: default
          database: uptrace

          tls:
            ca_file: /etc/clickhouse/certs/Uptrace_CA.crt
            cert_file: /etc/clickhouse/certs/chnode.crt
            key_file: /etc/clickhouse/certs/chnode.key
            insecure_skip_verify: false

ClickHouse Cloud Configuration:

yml
ch_cluster:
  shards:
    - replicas:
        - addr: tm849a32za.us-central1.gcp.clickhouse.cloud:9440
          user: default
          password: your_cloud_password
          database: uptrace
          tls:
            # Override server name if address doesn't match certificate
            server_name_override: 'tm849a32za.us-central1.gcp.clickhouse.cloud'

Schema Configuration and Optimization

Customize ClickHouse schema settings for optimal performance and storage efficiency.

⚠️ Critical: Schema changes require database reset and will delete all existing data:

shell
uptrace ch reset

Storage and Compression Settings

yml
ch_schema:
  # Compression algorithm selection
  # LZ4: Fast compression/decompression, moderate ratio
  # ZSTD(1): Better compression ratio, slightly slower
  # Default: ClickHouse default compression
  compression: ZSTD(1)

  # Storage policies for different data types
  # Configure different storage tiers (SSD, HDD, S3) based on access patterns
  spans_index: { storage_policy: hot_storage }    # Frequently accessed span indexes
  spans_data: { storage_policy: warm_storage }    # Span detail data
  span_links: { storage_policy: cold_storage }    # Span relationship data
  logs_index: { storage_policy: hot_storage }     # Log search indexes
  logs_data: { storage_policy: warm_storage }     # Log detail data
  events_index: { storage_policy: hot_storage }   # Event search indexes
  events_data: { storage_policy: warm_storage }   # Event detail data
  metrics: { storage_policy: hot_storage }        # Metrics time-series data

ClickHouse Replication Setup

Step 1: Configure ClickHouse Cluster

Create a cluster configuration with at least 3 replicas. Critical: Set internal_replication = true to prevent data duplication.

xml
<clickhouse>
  <remote_servers>
    <uptrace1>
      <shard>
        <internal_replication>true</internal_replication>
        <replica>
          <host>clickhouse-1</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>clickhouse-2</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>clickhouse-3</host>
          <port>9000</port>
        </replica>
      </shard>
    </uptrace1>
  </remote_servers>
</clickhouse>

Step 2: Enable Replication in Uptrace

yml
ch_cluster:
  cluster: 'uptrace1'
  replicated: true  # Enable ClickHouse table replication

  shards:
    - replicas:
        - addr: clickhouse-1:9000
        - addr: clickhouse-2:9000
        - addr: clickhouse-3:9000

Step 3: Apply Configuration Changes

shell
uptrace ch reset

Step 4: Verify Replication Status

sql
SELECT
    database,
    table,
    is_leader,
    replica_is_active
FROM system.replicas;

Expected output showing active replicas:

text
┌─database─┬─table───────────┬─is_leader─┬─replica_is_active────────────────────────┐
│ uptrace  │ spans_data      │         1 │ {'replica1':1,'replica2':1,'replica3':1} │
│ uptrace  │ spans_index     │         1 │ {'replica1':1,'replica2':1,'replica3':1} │
└──────────┴─────────────────┴───────────┴──────────────────────────────────────────┘

Network and Domain Configuration

Port Configuration

Configure network listeners for different protocols:

yml
listen:
  # HTTP server: OTLP/HTTP API, REST API, and web interface
  http:
    addr: ':80'  # Standard HTTP port

  # gRPC server: OTLP/gRPC for high-performance telemetry ingestion
  grpc:
    addr: ':4317'  # Standard OTLP gRPC port

  # TLS configuration for both HTTP and gRPC
  #tls:
  #  cert_file: /etc/uptrace/uptrace.crt
  #  key_file: /etc/uptrace/uptrace.key

Domain and URL Configuration

Primary Site Configuration:

yml
site:
  # Primary URL for user interface and API access
  # Used for: dashboard links, CORS validation, authentication redirects
  # Must be accessible from all clients and match reverse proxy setup
  url: https://uptrace.mycompany.com

Separate Ingestion Endpoint:

yml
site:
  url: https://uptrace.mycompany.com

  # Dedicated URL for telemetry data ingestion
  # Benefits: load balancing, security policies, CDN optimization
  # Defaults to site.url if not specified
  ingest_url: https://ingest.mycompany.com

Security and TLS Configuration

Automatic TLS with Let's Encrypt

For production deployments with automatic certificate management:

yml
##
## Automatic TLS certificate management via Let's Encrypt
##
certmagic:
  enabled: true                 # Enable automatic certificate issuance
  staging_ca: false            # Use production Let's Encrypt (set true for testing)
  http_challenge_addr: ':80'   # Port for HTTP-01 challenge validation

Manual TLS Certificate Configuration

Generate Self-Signed Certificate (Development):

shell
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout uptrace.key -out uptrace.crt -subj "/CN=localhost" \
  -addext "subjectAltName=DNS:localhost,DNS:uptrace.local"

Add to System Trust Store:

shell
sudo cp uptrace.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

Configure Uptrace with Custom Certificates:

yml
listen:
  grpc:
    addr: ':4317'
  http:
    addr: ':443'
  tls:
    cert_file: /etc/uptrace/tls/uptrace.crt
    key_file: /etc/uptrace/tls/uptrace.key
    min_version: '1.2'  # Minimum TLS version
    #ca_file: /etc/uptrace/tls/ca.crt  # Custom CA if needed

site:
  url: 'https://uptrace.mycompany.com'

Database TLS Configuration

Complete TLS Client Options:

yml
# PostgreSQL TLS example
pg:
  addr: localhost:5432
  user: uptrace
  password: uptrace
  database: uptrace
  tls:
    # Certificate files (optional for client authentication)
    cert_file: /etc/uptrace/client.crt
    key_file: /etc/uptrace/client.key
    ca_file: /etc/uptrace/ca.crt

    # TLS version constraints
    min_version: '1.2'
    max_version: '1.3'

    # Security settings
    insecure_skip_verify: false  # Verify server certificates
    server_name_override: ''     # Override server name verification

Common TLS Configurations:

yml
# Use system certificate store
pg:
  tls: {}

# Skip certificate verification (insecure, testing only)
pg:
  tls:
    insecure_skip_verify: true

# Disable TLS entirely
pg:
  tls:
    insecure: true

User Management and Authentication

Initial Setup Configuration

Customize default users and organizational structure:

yml
##
## Bootstrap data created on first startup
## Modify before first run - changes afterward require manual management
##
seed_data:
  users:
    - key: admin_user
      name: System Administrator
      email: admin@mycompany.com
      password: ChangeThisPassword123!  # Use strong password
      email_confirmed: true

  user_tokens:
    - key: admin_token
      user: admin_user
      token: secure_admin_token_here    # Generate random token

  orgs:
    - key: main_org
      name: My Company

  org_users:
    - key: admin_org_membership
      org: main_org
      user: admin_user
      role: owner                       # Roles: owner, admin, member

  projects:
    - key: production_project
      name: Production Environment
      org: main_org

  project_tokens:
    - key: prod_token
      project: production_project
      token: prod_telemetry_secret      # Used in OTLP DSN

  project_users:
    - key: admin_project_access
      project: production_project
      user: admin_user
      perm_level: admin                 # Levels: admin, editor, viewer

Authentication Controls

yml
auth:
  # Disable username/password authentication (SSO-only mode)
  # SSO providers remain available when disabled
  disabled: false

  # Restrict registration by email domain
  # Examples:
  # - Single domain: '^.+@mycompany\.com$'
  # - Multiple domains: '^.+@(company1|company2)\.com$'
  # - Subdomain wildcard: '^.+@.*\.mycompany\.com$'
  email_regexp: '^.+@mycompany\.com$'

Single Sign-On Integration

Uptrace supports enterprise identity providers:

Data Processing and Performance Tuning

Telemetry Data Processing Configuration

Optimize processing for different telemetry data types based on your ingestion volume:

yml
##
## Spans (distributed tracing data)
##
spans:
  # Processing parallelism (default: number of CPU cores)
  # Increase for high-volume tracing workloads
  max_threads: 16

  # Database insertion batch size
  # Larger batches improve throughput but increase memory usage
  max_insert_size: 20000

  # In-memory buffer capacity
  # Spans are dropped when buffer is full - monitor for drops
  max_buffered_records: 200000

##
## Logs (application log data)
##
logs:
  max_threads: 12
  max_insert_size: 15000
  max_buffered_records: 150000

##
## Metrics (time-series data)
##
metrics:
  max_threads: 8
  max_insert_size: 10000
  max_buffered_records: 100000

  # Cumulative to delta conversion capacity
  # Affects OpenTelemetry cumulative counter processing
  max_cumulative_timeseries: 2000000

##
## Events (custom application events)
##
events:
  max_threads: 4
  max_insert_size: 5000
  max_buffered_records: 50000

##
## Span Links (cross-trace relationships)
##
span_links:
  # Disable if not using span links to save resources
  disabled: false
  max_threads: 2
  max_insert_size: 5000
  max_buffered_records: 25000

Query Performance Limits

Prevent resource exhaustion from expensive queries:

yml
##
## Trace query resource limits
##
trace:
  # Maximum spans returned per query
  # Prevents UI timeouts and excessive memory usage
  query_limit: 500000

  # Maximum memory per query (bytes)
  # Prevents out-of-memory errors
  max_memory_usage_bytes: 500000000  # 500MB

Caching Configuration

Redis caching improves query performance and reduces database load:

yml
##
## Redis cache for query performance optimization
##
redis_cache:
  # Redis server addresses (cluster support)
  addrs:
    1: redis-1:6379
    2: redis-2:6379

  # Authentication
  username: ''
  password: 'redis_password'
  db: 0

  # Connection settings
  dial_timeout: 5s
  read_timeout: 3s
  write_timeout: 3s

  # TLS for secure connections
  #tls:
  #  insecure_skip_verify: false

Feature Management

Optional Feature Modules

Enable or disable features based on your needs:

yml
##
## Monitoring and alerting system
##
alerting:
  # Disable to save resources if not using alerts
  disabled: false

##
## Service dependency graph generation
##
service_graph:
  # Disable to save CPU/memory if not using service maps
  disabled: false

##
## JavaScript sourcemap processing
##
sourcemaps:
  # Disable in air-gapped environments or if not tracking JS errors
  # Requires internet access to download sourcemaps
  disabled: false

##
## Internal telemetry collection
##
monitoring:
  # Disable self-monitoring to prevent internal telemetry data
  disabled: false
  # DSN for sending Uptrace's own telemetry
  dsn: http://project1_secret@uptrace.local/2?grpc=4317

Notification and Communication

Email Configuration

Configure SMTP for alert notifications and user management:

yml
##
## SMTP mailer configuration
## Required for: alerts, password resets, user invitations
##
mailer:
  smtp:
    enabled: true
    host: smtp.mycompany.com
    port: 587                    # Common ports: 25, 465, 587
    username: uptrace@mycompany.com
    password: smtp_password
    from: 'Uptrace Alerts <uptrace@mycompany.com>'

    # TLS settings
    #tls:
    #  disabled: false           # Enable TLS (recommended)

Gmail Configuration Example:

  1. Generate app-specific password in Gmail settings
  2. Configure Uptrace:
yml
mailer:
  smtp:
    enabled: true
    host: smtp.gmail.com
    port: 587
    username: youraccount@gmail.com
    password: your_app_specific_password  # Not your regular password
    from: 'youraccount@gmail.com'

Telegram Integration

Enable Telegram notifications for alerts:

yml
##
## Telegram bot configuration
## Setup guide: https://core.telegram.org/bots#6-botfather
##
telegram:
  # Bot token from @BotFather
  bot_token: 'your_telegram_bot_token'

System Configuration

Application Logging

Control Uptrace's internal logging:

yml
##
## Application logging configuration
##
logging:
  # Log levels: DEBUG, INFO, WARN, ERROR
  # DEBUG: Verbose output for troubleshooting
  # INFO: Standard operational information
  # WARN: Warnings and errors only
  # ERROR: Errors only
  level: INFO

Premium Features

Enable advanced features with a license:

yml
##
## Premium Edition license
## Enables: distributed tables, advanced SSO, priority support
##
license:
  key: 'your_premium_license_key'

Premium Features Include:

  • ClickHouse distributed tables for horizontal scaling
  • Advanced SSO integrations
  • Priority technical support
  • Enhanced security features

Deployment Patterns

Production Checklist

Security Requirements:

  • ✅ Change all default passwords and tokens in seed_data
  • ✅ Configure TLS/SSL certificates for all connections
  • ✅ Set up authentication restrictions (auth.email_regexp)
  • ✅ Enable secure database connections
  • ✅ Configure proper firewall rules

High Availability Setup:

  • ✅ Deploy multiple Uptrace instances behind load balancer
  • ✅ Configure PostgreSQL with read replicas
  • ✅ Set up ClickHouse replication (minimum 3 nodes)
  • ✅ Configure Redis for caching and session storage
  • ✅ Set up database backup procedures

Performance Optimization:

  • ✅ Tune telemetry processing settings based on volume
  • ✅ Configure appropriate resource limits
  • ✅ Set up monitoring for Uptrace itself
  • ✅ Configure data retention policies
  • ✅ Optimize ClickHouse storage policies

Operational Requirements:

  • ✅ Configure email notifications for alerts
  • ✅ Set up log aggregation for Uptrace logs
  • ✅ Document recovery procedures
  • ✅ Configure backup verification
  • ✅ Set up monitoring dashboards

Reverse Proxy Configuration

When deploying behind a reverse proxy, ensure proper URL configuration:

yml
site:
  # Main deployment
  url: https://uptrace.mycompany.com

  # Subpath deployment
  url: https://mycompany.com/uptrace

Load Balancer Health Checks:

  • HTTP endpoint: GET /api/health
  • Expected response: 200 OK

Scaling Guidelines

Horizontal Scaling:

  • Uptrace instances are stateless and can be scaled horizontally
  • Use load balancer to distribute traffic across instances
  • Each instance should connect to the same PostgreSQL and ClickHouse databases

Vertical Scaling Recommendations: According to ClickHouse documentation, prefer vertical scaling over horizontal scaling:

Successful deployments with ClickHouse use servers with hundreds of cores, terabytes of RAM, and petabytes of disk space. Scaling vertically first provides cost efficiency, lower operational overhead, and better query performance.

Resource Requirements:

  • Uptrace instances: 2-4 CPU cores, 4-8GB RAM per instance
  • PostgreSQL: 2-4 CPU cores, 4-8GB RAM, SSD storage
  • ClickHouse: Scale based on data volume (prefer large instances)
  • Redis: 1-2 CPU cores, 2-4GB RAM

Troubleshooting Guide

Common Configuration Issues

Database Connection Problems:

yml
# Check connection settings
pg:
  addr: localhost:5432  # Verify host and port
  user: uptrace         # Ensure user exists
  password: uptrace     # Verify password
  database: uptrace     # Ensure database exists

TLS Certificate Issues:

bash
# Verify certificate validity
openssl x509 -in uptrace.crt -text -noout

# Check certificate expiration
openssl x509 -in uptrace.crt -checkend 86400

# Test TLS connection
openssl s_client -connect uptrace.local:443

Performance Problems:

  • Monitor ClickHouse resource usage
  • Check for dropped telemetry data in metrics
  • Review query performance in ClickHouse logs
  • Verify network latency between components

Authentication Issues:

  • Verify email regex patterns if configured
  • Check SSO provider configuration
  • Confirm user permissions and project access
  • Review authentication logs for error details

Monitoring Uptrace Health

Key Metrics to Monitor:

  • Database connection health
  • Telemetry data ingestion rates
  • Query response times
  • Memory and CPU usage
  • Disk space utilization

Health Check Endpoints:

  • API health: GET /api/health
  • Database connectivity: Monitor connection metrics
  • Redis connectivity: Monitor cache hit rates