Uptrace Configuration Guide

Uptrace is an open-source APM and observability platform that supports distributed tracing, metrics, and log management. You can customize all Uptrace settings and modify the generated ClickHouse database schema using a single YAML configuration file.

Configuration File Overview

All Uptrace configuration is managed through a single YAML file. You can create the default configuration file using the Uptrace binary:

shell
./uptrace -config=/path/to/config.yml config create

Configuration File Location

You can specify the configuration file location in two ways:

Using a CLI argument:

shell
uptrace --config=/etc/uptrace/config.yml serve

Using an environment variable:

shell
UPTRACE_CONFIG=/etc/uptrace/config.yml uptrace serve

If you don't explicitly specify the configuration location, Uptrace defaults to /etc/uptrace/config.yml.

Database Configuration

Uptrace requires two databases: PostgreSQL for metadata storage and ClickHouse for telemetry data.

PostgreSQL Configuration

Uptrace requires a PostgreSQL database to store metadata such as metric names, alerts, and user information. This database is typically small, occupying only a few megabytes of disk space.

Configure PostgreSQL database credentials in your config file:

yaml
##
## PostgreSQL database configuration
##
pg:
  addr: localhost:5432
  user: uptrace
  password: uptrace
  database: uptrace

  # TLS configuration (uncomment to enable)
  #tls:
  #  insecure_skip_verify: true # only for self-signed certificates

High Availability Setup:

For high availability, you can configure PostgreSQL standby nodes:

yaml
pg:
  addr: primary-host:5432
  user: uptrace
  password: uptrace
  database: uptrace

  # Standby replicas for read operations
  replicas:
    - standby-host1:5432
    - standby-host2:5432

See the TLS Client Configuration section for SSL certificate configuration.

ClickHouse Configuration

ClickHouse is the primary storage engine for telemetry data including spans, logs, and metrics. Configure ClickHouse database credentials as follows:

yaml
ch_cluster:
  cluster: 'uptrace1'

  # Enable ClickHouse replication for high availability
  # Cluster name is required when replication is enabled
  replicated: false

  # Enable ClickHouse distributed tables for horizontal scaling
  # Requires Premium Edition
  distributed: false

  shards:
    - replicas:
        - addr: localhost:9000
          user: default
          password: ''
          database: uptrace

High Availability Setup:

For high availability, configure multiple ClickHouse replicas:

yaml
ch_cluster:
  cluster: 'uptrace1'
  replicated: true

  shards:
    - replicas:
        - addr: replica1:9000
          user: default
          password: ''
          database: uptrace

        - addr: replica2:9000
          user: default
          password: ''
          database: uptrace

        - addr: replica3:9000
          user: default
          password: ''
          database: uptrace

Environment Variables

You can use environment variables in your YAML configuration file to keep sensitive information secure:

yaml
pg:
  addr: ${UPTRACE_PG_ADDR:localhost:5432}
  user: ${UPTRACE_PG_USER:uptrace}
  password: ${UPTRACE_PG_PASSWORD}
  database: ${UPTRACE_PG_DATABASE:uptrace}

Environment Variable Format:

  • Simple substitution: ${ENV_VAR_NAME}
  • With default values: ${ENV_VAR_NAME:default_value}
  • To escape $: Use $$

Variables are expanded before parsing the YAML content using Go's os.Expand function.

ClickHouse Advanced Configuration

Query Settings

To modify ClickHouse query settings:

yaml
ch_cluster:
  shards:
    - replicas:
        - addr: localhost:9000
          query_settings:
            async_insert: 1 # enabled by default
            wait_for_async_insert: 1 # disabled by default

TLS Configuration

To use TLS connections with ClickHouse, you need to enable the secure TCP port (9440) and configure SSL certificates in your ClickHouse configuration:

ClickHouse Server Configuration:

xml
<?xml version="1.0" ?>
<clickhouse>
  <tcp_port_secure>9440</tcp_port_secure>

  <openSSL>
    <server>
      <certificateFile>/etc/clickhouse/certs/chnode.crt</certificateFile>
      <privateKeyFile>/etc/clickhouse/certs/chnode.key</privateKeyFile>
      <verificationMode>relaxed</verificationMode>
      <caConfig>/etc/clickhouse/certs/Uptrace_CA.crt</caConfig>
      <cacheSessions>true</cacheSessions>
      <disableProtocols>sslv2,sslv3</disableProtocols>
      <preferServerCiphers>true</preferServerCiphers>
    </server>
  </openSSL>
</clickhouse>

Uptrace TLS Configuration:

yaml
ch_cluster:
  shards:
    - replicas:
        - addr: localhost:9440
          user: default
          database: uptrace

          tls:
            ca_file: /etc/clickhouse/certs/Uptrace_CA.crt
            cert_file: /etc/clickhouse/certs/chnode.crt
            key_file: /etc/clickhouse/certs/chnode.key
            insecure_skip_verify: true

ClickHouse Cloud Configuration:

For ClickHouse Cloud, you may need to override the server name if the address doesn't match the certificate:

yaml
ch_cluster:
  shards:
    - replicas:
        - addr: tm849a32za.us-central1.gcp.clickhouse.cloud:9440
          tls:
            server_name_override: 'tm849a32za.us-central1.gcp.clickhouse.cloud'

Performance Considerations

Uptrace performance is primarily limited by ClickHouse database performance. When scaling ClickHouse, prefer vertical scaling over horizontal scaling.

According to the ClickHouse blog:

We commonly find successful deployments with ClickHouse deployed on servers with hundreds of cores, terabytes of RAM, and petabytes of disk space.

Scaling vertically first has a number of benefits, principally cost efficiency, lower cost of ownership (with respect to operations), and better query performance due to the minimization of data on the network for operations such as JOINs.

ClickHouse Schema Configuration

The following options allow you to modify the ClickHouse schema generated by Uptrace.

⚠️ Important: For changes to take effect, you must reset the ClickHouse database:

shell
uptrace ch reset

Compression Configuration

Configure the compression method for ClickHouse tables:

yaml
ch_schema:
  # Compression codec options: LZ4, ZSTD(1), or Default
  compression: ZSTD(1)

  # Storage policies for different data types
  spans_index:
    storage_policy: default
  spans_data:
    storage_policy: default
  logs_index:
    storage_policy: default
  logs_data:
    storage_policy: default
  metrics:
    storage_policy: default

Replication Setup

To enable ClickHouse table replication, follow these steps:

Step 1: Configure ClickHouse Cluster

Configure a ClickHouse cluster with at least 3 replicas. Important: Keep internal_replication = true to avoid data duplicates.

xml
<clickhouse>
  <remote_servers>
    <uptrace1>
      <shard>
        <internal_replication>true</internal_replication>
        <replica>
          <host>clickhouse-1</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>clickhouse-2</host>
          <port>9000</port>
        </replica>
        <replica>
          <host>clickhouse-3</host>
          <port>9000</port>
        </replica>
      </shard>
    </uptrace1>
  </remote_servers>
</clickhouse>

Step 2: Update Uptrace Configuration

Update your Uptrace configuration to use the cluster name:

yaml
ch_cluster:
  cluster: 'uptrace1'
  # Enable ClickHouse replication
  replicated: true

  shards:
    - replicas:
        - addr: clickhouse-1:9000
        - addr: clickhouse-2:9000
        - addr: clickhouse-3:9000

Step 3: Reset ClickHouse Database

Apply the changes by resetting the ClickHouse database:

shell
uptrace ch reset

Step 4: Verify Replication

Verify replication is working using clickhouse-client:

sql
SELECT
    database,
    table,
    is_leader,
    replica_is_active
FROM system.replicas;

Expected output:

text
┌─database─┬─table───────────┬─is_leader─┬─replica_is_active────────────────────────┐
│ uptrace  │ spans_data      │         1 │ {'replica1':1,'replica2':1,'replica3':1} │
│ uptrace  │ spans_index     │         1 │ {'replica1':1,'replica2':1,'replica3':1} │
└──────────┴─────────────────┴───────────┴──────────────────────────────────────────┘

Network Configuration

Port Configuration

By default, Uptrace listens on the following ports:

  • Port 4317 for OTLP/gRPC
  • Port 80 for OTLP/HTTP and web UI

You can customize these ports in your configuration:

yaml
listen:
  # HTTP server for OTLP/HTTP API and Vue.js UI
  http:
    addr: ':80'

  # gRPC server for OTLP/gRPC API
  grpc:
    addr: ':4317'

  # TLS configuration for both HTTP and gRPC (uncomment to enable)
  #tls:
  #  cert_file: /etc/uptrace/uptrace.crt
  #  key_file: /etc/uptrace/uptrace.key

Domain Configuration

Configure the Uptrace domain for the Vue-powered UI and DSN/endpoints:

yaml
site:
  # Public URL for the UI and DSN endpoint
  # Port should match listen.http.addr unless using a reverse proxy
  url: 'https://uptrace.mydomain.com'

After making changes, restart Uptrace:

shell
sudo systemctl restart uptrace

TLS/SSL Configuration

Uptrace supports multiple TLS configuration options for secure connections.

Automatic TLS with Let's Encrypt

For HTTP and gRPC listeners, Uptrace supports automatic Let's Encrypt certificates using the certmagic library:

yaml
##
## Automatic TLS certificate issuance and renewal using Let's Encrypt
##
certmagic:
  # Enable Let's Encrypt certificates
  enabled: true
  # Use Let's Encrypt staging environment (for testing)
  staging_ca: false
  # Address for HTTP-01 challenge validation
  http_challenge_addr: ':80'

Manual TLS Server Configuration

For custom certificates, first generate a certificate (replace localhost with your domain):

shell
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
  -keyout uptrace.key -out uptrace.crt -subj "/CN=localhost" \
  -addext "subjectAltName=DNS:localhost"

Add the certificate to trusted certificates:

shell
sudo cp uptrace.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates

Configure Uptrace to use the certificate:

yaml
listen:
  grpc:
    addr: ':4317'
  http:
    addr: ':443'
  tls:
    cert_file: config/tls/uptrace.crt
    key_file: config/tls/uptrace.key
    #ca_file: path/to/ca_file

site:
  # Update domain for Vue-powered UI
  url: 'https://uptrace.mydomain.com'

Complete TLS server options:

yaml
listen:
  tls:
    # Path to CA certificate (optional)
    ca_file: ''
    # Path to TLS certificate
    cert_file: ''
    # Path to TLS private key
    key_file: ''
    # Minimum TLS version (default: TLS 1.2)
    min_version: ''
    # Maximum TLS version (optional)
    max_version: ''
    # Client certificate verification
    client_ca_file: ''

TLS Client Configuration

For database connections, you can configure TLS in several ways:

Use system certificates:

yaml
pg:
  tls: {}

Skip certificate verification (insecure):

yaml
pg:
  tls:
    insecure_skip_verify: true

Use custom certificates:

yaml
pg:
  tls:
    cert_file: path/to/client.crt
    key_file: path/to/client.key
    ca_file: path/to/ca.crt

Disable TLS:

yaml
pg:
  tls:
    insecure: true

Override server name:

yaml
ch_cluster:
  shards:
    - replicas:
        - addr: tm849a32za.us-central1.gcp.clickhouse.cloud:9440
          tls:
            server_name_override: 'tm849a32za.us-central1.gcp.clickhouse.cloud'

Complete TLS client options:

yaml
tls:
  # Disable client transport security (default: false)
  insecure: false
  # Path to CA certificate (optional)
  ca_file: ''
  # Path to client certificate (optional)
  cert_file: ''
  # Path to client private key (optional)
  key_file: ''
  # Minimum TLS version (default: TLS 1.2)
  min_version: ''
  # Maximum TLS version (optional)
  max_version: ''
  # Skip certificate verification (default: false)
  insecure_skip_verify: false
  # Override server name (optional)
  server_name_override: ''

User Authentication and Initial Setup

Initial Data Configuration

On first startup, Uptrace creates a default user with login admin@uptrace.local and password admin. You can customize this initial data in the seed_data section:

yaml
# Initial data created on first startup
seed_data:
  users:
    - name: Admin
      key: user1 # Unique identifier for referencing in relationships
      email: admin@uptrace.local
      password: admin # TODO: Change default password in production!
      auth_tokens: [user1_secret]

  orgs:
    - name: Default Organization
      key: org1 # Unique identifier for referencing below

  org_users:
    - org: org1 # References org.key above
      user: user1 # References user.key above
      role: owner

  projects:
    - name: Default Project
      key: project1 # Unique identifier for referencing below
      org: org1 # References org.key above
      tokens: [project1_secret]

  project_users:
    - project: project1 # References project.key above
      user: user1 # References user.key above
      perm_level: admin

Authentication Configuration

Control user registration and authentication:

yaml
auth:
  # Set to true to disable username/password authentication
  # Note: SSO authentication will remain available
  disabled: false

  # Email address validation pattern (optional)
  # Only users with matching email addresses can register/login
  # Example: '^.+@example\.com$' restricts to @example.com domain
  email_regexp: '^.+@example\.com$'

Single Sign-On (SSO) Integration

Uptrace supports integration with popular identity providers:

Email Notifications

To enable email notifications for alerts, configure an SMTP mailer:

yaml
##
## SMTP configuration for email notifications
## Documentation: https://uptrace.dev/features/alerting
##
mailer:
  smtp:
    # Enable this mailer for sending emails
    enabled: true
    # SMTP server host
    host: localhost
    # SMTP server port
    port: 1025
    # Authentication username
    username: mailhog
    # Authentication password
    password: mailhog
    # Sender email address
    from: 'uptrace@localhost'

    # TLS configuration (uncomment to disable opportunistic TLS)
    #tls:
    #  disabled: true

Gmail Configuration

Gmail requires an app-specific password instead of your regular password:

  1. In Gmail, click your avatar → "Manage your Google Account"
  2. Navigate to "Security" in the left sidebar
  3. Scroll to "Signing in to Google" and click "App password"
  4. Generate a new app password for Uptrace

See Gmail's documentation for detailed instructions.

Gmail SMTP Configuration:

yaml
mailer:
  smtp:
    enabled: true
    host: smtp.gmail.com
    port: 587
    username: your-email@gmail.com
    password: your-app-password
    from: 'your-email@gmail.com'

Caching Configuration

Uptrace uses Redis for caching to improve query performance:

yaml
redis_cache:
  addrs:
    1: localhost:6379

  # Redis authentication
  username: ''
  password: ''
  db: 0

  # TLS configuration (uncomment for secure connections)
  #tls:
  #  insecure_skip_verify: true

Reverse Proxy Configuration

When running Uptrace behind a reverse proxy (Nginx, HAProxy, etc.), configure the domain name so Uptrace can render links and redirects correctly:

yaml
site:
  url: 'https://uptrace.mydomain.com'

Subpath deployment (e.g., http://mydomain.com/uptrace):

yaml
site:
  url: 'https://mydomain.com/uptrace'

Scaling and High Availability

Uptrace is stateless and can be scaled horizontally by launching multiple nodes and load balancing traffic.

Default Ports

ServicePort
OTLP HTTP and API80
OTLP gRPC4317

High Availability Setup

For high availability:

  1. Deploy multiple Uptrace instances behind a load balancer
  2. Configure PostgreSQL with standby replicas for metadata redundancy
  3. Set up ClickHouse replication for telemetry data redundancy
  4. Use a load balancer (HAProxy, Nginx) to distribute traffic across instances

Premium Features

License Configuration

For Premium Edition features, specify your Uptrace license key:

yaml
license:
  key: '<YOUR_LICENSE_KEY>'

Distributed Tables

Distributed tables for horizontal scaling (requires Premium Edition):

yaml
ch_cluster:
  # Enable ClickHouse distributed tables for horizontal scaling
  # Requires Premium Edition
  distributed: true

Quick Start Guide

Initial Setup

Follow these steps to get Uptrace running:

  1. Create configuration file:
    shell
    ./uptrace -config=/etc/uptrace/config.yml config create
    
  2. Configure databases:
    • Set PostgreSQL credentials
    • Set ClickHouse credentials
  3. Configure basic settings:
    • Set site URL
    • Configure ports (if needed)
    • Set up TLS (recommended for production)
  4. Start Uptrace:
    shell
    uptrace --config=/etc/uptrace/config.yml serve
    
  5. Access the UI:
    • Default login: admin@uptrace.local
    • Default password: admin

Production Deployment Checklist

For production deployments, ensure you:

  • ✅ Change default credentials in seed_data
  • ✅ Configure TLS/SSL certificates
  • ✅ Set up proper database backups
  • ✅ Configure email notifications
  • ✅ Set up monitoring and alerting
  • ✅ Configure data retention policies
  • ✅ Enable authentication restrictions if needed
  • ✅ Set up high availability (multiple instances, database replication)
  • ✅ Configure reverse proxy if using one

Troubleshooting

Common Issues

Database connection errors:

  • Verify database credentials and connectivity
  • Check firewall rules for database ports
  • Ensure database services are running

TLS certificate errors:

  • Verify certificate paths and permissions
  • Check certificate expiration dates
  • Ensure certificate matches the configured domain

Performance issues:

  • Monitor ClickHouse performance and resources
  • Consider vertical scaling for ClickHouse
  • Review query performance and optimization

Authentication problems:

  • Check email regexp patterns if configured
  • Verify SSO configuration if using external providers
  • Ensure proper permissions are set for users and projects