Uptrace Configuration Reference
Uptrace is an open-source APM and observability platform that supports distributed tracing, metrics, and log management. This guide covers comprehensive configuration options for customizing Uptrace settings and optimizing the ClickHouse database schema through a single YAML configuration file.
Configuration File Management
Creating and Locating Configuration Files
Generate a default configuration file using the Uptrace binary:
./uptrace config create -config=/path/to/config.yml
Specify the configuration file location using either method:
CLI argument (recommended):
uptrace --config=/etc/uptrace/config.yml serve
Environment variable:
export UPTRACE_CONFIG=/etc/uptrace/config.yml
uptrace serve
If no configuration path is specified, Uptrace defaults to /etc/uptrace/config.yml
.
Environment Variable Integration
Enhance security and flexibility by using environment variables in your YAML configuration:
pg:
addr: ${UPTRACE_PG_ADDR:localhost:5432}
user: ${UPTRACE_PG_USER:uptrace}
password: ${UPTRACE_PG_PASSWORD}
database: ${UPTRACE_PG_DATABASE:uptrace}
Environment Variable Syntax:
- Simple substitution:
${ENV_VAR_NAME}
- With fallback defaults:
${ENV_VAR_NAME:default_value}
- Escape dollar signs: Use
$$
to literal$
Variables are expanded using Go's os.Expand function before YAML parsing.
Database Architecture and Configuration
Uptrace uses a dual-database architecture optimized for different data types and access patterns.
PostgreSQL: Metadata Storage
PostgreSQL stores application metadata including user accounts, project settings, dashboards, and alert configurations. This database typically requires only a few megabytes of storage.
Basic Configuration:
pg:
addr: localhost:5432
user: uptrace
password: uptrace
database: uptrace
# TLS configuration for secure connections
# Recommended for production deployments
#tls:
# insecure_skip_verify: true # WARNING: Only for self-signed certificates
High Availability with Read Replicas:
pg:
addr: primary-host:5432
user: uptrace
password: uptrace
database: uptrace
# Standby replicas
replicas:
- standby-host1:5432
- standby-host2:5432
Using connection string:
pg:
dsn: postgresql://user:password@host:5432/database?sslmode=verify-full
ClickHouse: Telemetry Data Storage
ClickHouse serves as the primary storage engine for high-volume telemetry data including spans, logs, and metrics. It's optimized for analytical queries and time-series data.
Basic Configuration:
ch_cluster:
# Cluster name for ClickHouse operations
cluster: 'uptrace1'
# High availability features
replicated: false # Enable for automatic failover
distributed: false # Enable for horizontal scaling (Premium only)
shards:
- replicas:
- addr: localhost:9000
user: default
password: ''
database: uptrace
# Connection and query timeouts
dial_timeout: 3s
write_timeout: 5s
max_retries: 3
max_execution_time: 15s
Production High Availability Setup:
ch_cluster:
cluster: 'uptrace1'
replicated: true
shards:
- replicas:
- addr: clickhouse-1:9000
user: default
password: ''
database: uptrace
- addr: clickhouse-2:9000
user: default
password: ''
database: uptrace
- addr: clickhouse-3:9000
user: default
password: ''
database: uptrace
Using connection string:
ch_cluster:
shards:
- replicas:
- dsn: clickhouse://your_username:your_password@your_host:9000/your_database
# Connection and query timeouts
dial_timeout: 3s
write_timeout: 5s
max_retries: 3
max_execution_time: 15s
ClickHouse Advanced Configuration
Query Performance Tuning:
ch_cluster:
shards:
- replicas:
- addr: localhost:9000
# ClickHouse-specific query settings
query_settings:
async_insert: 1 # Enable async inserts (default: enabled)
wait_for_async_insert: 1 # Wait for insert completion (default: disabled)
max_memory_usage: 10000000000 # 10GB memory limit per query
TLS Configuration for ClickHouse:
First, configure ClickHouse server to enable secure TCP port:
<!-- ClickHouse server configuration -->
<?xml version="1.0" ?>
<clickhouse>
<tcp_port_secure>9440</tcp_port_secure>
<openSSL>
<server>
<certificateFile>/etc/clickhouse/certs/chnode.crt</certificateFile>
<privateKeyFile>/etc/clickhouse/certs/chnode.key</privateKeyFile>
<verificationMode>relaxed</verificationMode>
<caConfig>/etc/clickhouse/certs/Uptrace_CA.crt</caConfig>
<cacheSessions>true</cacheSessions>
<disableProtocols>sslv2,sslv3</disableProtocols>
<preferServerCiphers>true</preferServerCiphers>
</server>
</openSSL>
</clickhouse>
Then configure Uptrace to use TLS:
ch_cluster:
shards:
- replicas:
- addr: localhost:9440 # Use secure port
user: default
database: uptrace
tls:
ca_file: /etc/clickhouse/certs/Uptrace_CA.crt
cert_file: /etc/clickhouse/certs/chnode.crt
key_file: /etc/clickhouse/certs/chnode.key
insecure_skip_verify: false
ClickHouse Cloud Configuration:
ch_cluster:
shards:
- replicas:
- addr: tm849a32za.us-central1.gcp.clickhouse.cloud:9440
user: default
password: your_cloud_password
database: uptrace
tls:
# Override server name if address doesn't match certificate
server_name_override: 'tm849a32za.us-central1.gcp.clickhouse.cloud'
Schema Configuration and Optimization
Customize ClickHouse schema settings for optimal performance and storage efficiency.
⚠️ Critical: Schema changes require database reset and will delete all existing data:
uptrace ch reset
Storage and Compression Settings
ch_schema:
# Compression algorithm selection
# LZ4: Fast compression/decompression, moderate ratio
# ZSTD(1): Better compression ratio, slightly slower
# Default: ClickHouse default compression
compression: ZSTD(1)
# Storage policies for different data types
# Configure different storage tiers (SSD, HDD, S3) based on access patterns
spans_index: { storage_policy: hot_storage } # Frequently accessed span indexes
spans_data: { storage_policy: warm_storage } # Span detail data
span_links: { storage_policy: cold_storage } # Span relationship data
logs_index: { storage_policy: hot_storage } # Log search indexes
logs_data: { storage_policy: warm_storage } # Log detail data
events_index: { storage_policy: hot_storage } # Event search indexes
events_data: { storage_policy: warm_storage } # Event detail data
metrics: { storage_policy: hot_storage } # Metrics time-series data
ClickHouse Replication Setup
Step 1: Configure ClickHouse Cluster
Create a cluster configuration with at least 3 replicas. Critical: Set internal_replication = true
to prevent data duplication.
<clickhouse>
<remote_servers>
<uptrace1>
<shard>
<internal_replication>true</internal_replication>
<replica>
<host>clickhouse-1</host>
<port>9000</port>
</replica>
<replica>
<host>clickhouse-2</host>
<port>9000</port>
</replica>
<replica>
<host>clickhouse-3</host>
<port>9000</port>
</replica>
</shard>
</uptrace1>
</remote_servers>
</clickhouse>
Step 2: Enable Replication in Uptrace
ch_cluster:
cluster: 'uptrace1'
replicated: true # Enable ClickHouse table replication
shards:
- replicas:
- addr: clickhouse-1:9000
- addr: clickhouse-2:9000
- addr: clickhouse-3:9000
Step 3: Apply Configuration Changes
uptrace ch reset
Step 4: Verify Replication Status
SELECT
database,
table,
is_leader,
replica_is_active
FROM system.replicas;
Expected output showing active replicas:
┌─database─┬─table───────────┬─is_leader─┬─replica_is_active────────────────────────┐
│ uptrace │ spans_data │ 1 │ {'replica1':1,'replica2':1,'replica3':1} │
│ uptrace │ spans_index │ 1 │ {'replica1':1,'replica2':1,'replica3':1} │
└──────────┴─────────────────┴───────────┴──────────────────────────────────────────┘
Network and Domain Configuration
Port Configuration
Configure network listeners for different protocols:
listen:
# HTTP server: OTLP/HTTP API, REST API, and web interface
http:
addr: ':80' # Standard HTTP port
# gRPC server: OTLP/gRPC for high-performance telemetry ingestion
grpc:
addr: ':4317' # Standard OTLP gRPC port
# TLS configuration for both HTTP and gRPC
#tls:
# cert_file: /etc/uptrace/uptrace.crt
# key_file: /etc/uptrace/uptrace.key
Domain and URL Configuration
Primary Site Configuration:
site:
# Primary URL for user interface and API access
# Used for: dashboard links, CORS validation, authentication redirects
# Must be accessible from all clients and match reverse proxy setup
url: https://uptrace.mycompany.com
Separate Ingestion Endpoint:
site:
url: https://uptrace.mycompany.com
# Dedicated URL for telemetry data ingestion
# Benefits: load balancing, security policies, CDN optimization
# Defaults to site.url if not specified
ingest_url: https://ingest.mycompany.com
Security and TLS Configuration
Automatic TLS with Let's Encrypt
For production deployments with automatic certificate management:
##
## Automatic TLS certificate management via Let's Encrypt
##
certmagic:
enabled: true # Enable automatic certificate issuance
staging_ca: false # Use production Let's Encrypt (set true for testing)
http_challenge_addr: ':80' # Port for HTTP-01 challenge validation
Manual TLS Certificate Configuration
Generate Self-Signed Certificate (Development):
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
-keyout uptrace.key -out uptrace.crt -subj "/CN=localhost" \
-addext "subjectAltName=DNS:localhost,DNS:uptrace.local"
Add to System Trust Store:
sudo cp uptrace.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
Configure Uptrace with Custom Certificates:
listen:
grpc:
addr: ':4317'
http:
addr: ':443'
tls:
cert_file: /etc/uptrace/tls/uptrace.crt
key_file: /etc/uptrace/tls/uptrace.key
min_version: '1.2' # Minimum TLS version
#ca_file: /etc/uptrace/tls/ca.crt # Custom CA if needed
site:
url: 'https://uptrace.mycompany.com'
Database TLS Configuration
Complete TLS Client Options:
# PostgreSQL TLS example
pg:
addr: localhost:5432
user: uptrace
password: uptrace
database: uptrace
tls:
# Certificate files (optional for client authentication)
cert_file: /etc/uptrace/client.crt
key_file: /etc/uptrace/client.key
ca_file: /etc/uptrace/ca.crt
# TLS version constraints
min_version: '1.2'
max_version: '1.3'
# Security settings
insecure_skip_verify: false # Verify server certificates
server_name_override: '' # Override server name verification
Common TLS Configurations:
# Use system certificate store
pg:
tls: {}
# Skip certificate verification (insecure, testing only)
pg:
tls:
insecure_skip_verify: true
# Disable TLS entirely
pg:
tls:
insecure: true
User Management and Authentication
Initial Setup Configuration
Customize default users and organizational structure:
##
## Bootstrap data created on first startup
## Modify before first run - changes afterward require manual management
##
seed_data:
users:
- key: admin_user
name: System Administrator
email: admin@mycompany.com
password: ChangeThisPassword123! # Use strong password
email_confirmed: true
user_tokens:
- key: admin_token
user: admin_user
token: secure_admin_token_here # Generate random token
orgs:
- key: main_org
name: My Company
org_users:
- key: admin_org_membership
org: main_org
user: admin_user
role: owner # Roles: owner, admin, member
projects:
- key: production_project
name: Production Environment
org: main_org
project_tokens:
- key: prod_token
project: production_project
token: prod_telemetry_secret # Used in OTLP DSN
project_users:
- key: admin_project_access
project: production_project
user: admin_user
perm_level: admin # Levels: admin, editor, viewer
Authentication Controls
auth:
# Disable username/password authentication (SSO-only mode)
# SSO providers remain available when disabled
disabled: false
# Restrict registration by email domain
# Examples:
# - Single domain: '^.+@mycompany\.com$'
# - Multiple domains: '^.+@(company1|company2)\.com$'
# - Subdomain wildcard: '^.+@.*\.mycompany\.com$'
email_regexp: '^.+@mycompany\.com$'
Single Sign-On Integration
Uptrace supports enterprise identity providers:
- Okta - Enterprise SSO platform
- Keycloak - Open-source identity management
- Cloudflare Access - Zero Trust security
- Google Workspace - Google enterprise accounts
Data Processing and Performance Tuning
Telemetry Data Processing Configuration
Optimize processing for different telemetry data types based on your ingestion volume:
##
## Spans (distributed tracing data)
##
spans:
# Processing parallelism (default: number of CPU cores)
# Increase for high-volume tracing workloads
max_threads: 16
# Database insertion batch size
# Larger batches improve throughput but increase memory usage
max_insert_size: 20000
# In-memory buffer capacity
# Spans are dropped when buffer is full - monitor for drops
max_buffered_records: 200000
##
## Logs (application log data)
##
logs:
max_threads: 12
max_insert_size: 15000
max_buffered_records: 150000
##
## Metrics (time-series data)
##
metrics:
max_threads: 8
max_insert_size: 10000
max_buffered_records: 100000
# Cumulative to delta conversion capacity
# Affects OpenTelemetry cumulative counter processing
max_cumulative_timeseries: 2000000
##
## Events (custom application events)
##
events:
max_threads: 4
max_insert_size: 5000
max_buffered_records: 50000
##
## Span Links (cross-trace relationships)
##
span_links:
# Disable if not using span links to save resources
disabled: false
max_threads: 2
max_insert_size: 5000
max_buffered_records: 25000
Query Performance Limits
Prevent resource exhaustion from expensive queries:
##
## Trace query resource limits
##
trace:
# Maximum spans returned per query
# Prevents UI timeouts and excessive memory usage
query_limit: 500000
# Maximum memory per query (bytes)
# Prevents out-of-memory errors
max_memory_usage_bytes: 500000000 # 500MB
Caching Configuration
Redis caching improves query performance and reduces database load:
##
## Redis cache for query performance optimization
##
redis_cache:
# Redis server addresses (cluster support)
addrs:
1: redis-1:6379
2: redis-2:6379
# Authentication
username: ''
password: 'redis_password'
db: 0
# Connection settings
dial_timeout: 5s
read_timeout: 3s
write_timeout: 3s
# TLS for secure connections
#tls:
# insecure_skip_verify: false
Feature Management
Optional Feature Modules
Enable or disable features based on your needs:
##
## Monitoring and alerting system
##
alerting:
# Disable to save resources if not using alerts
disabled: false
##
## Service dependency graph generation
##
service_graph:
# Disable to save CPU/memory if not using service maps
disabled: false
##
## JavaScript sourcemap processing
##
sourcemaps:
# Disable in air-gapped environments or if not tracking JS errors
# Requires internet access to download sourcemaps
disabled: false
##
## Internal telemetry collection
##
monitoring:
# Disable self-monitoring to prevent internal telemetry data
disabled: false
# DSN for sending Uptrace's own telemetry
dsn: http://project1_secret@uptrace.local/2?grpc=4317
Notification and Communication
Email Configuration
Configure SMTP for alert notifications and user management:
##
## SMTP mailer configuration
## Required for: alerts, password resets, user invitations
##
mailer:
smtp:
enabled: true
host: smtp.mycompany.com
port: 587 # Common ports: 25, 465, 587
username: uptrace@mycompany.com
password: smtp_password
from: 'Uptrace Alerts <uptrace@mycompany.com>'
# TLS settings
#tls:
# disabled: false # Enable TLS (recommended)
Gmail Configuration Example:
- Generate app-specific password in Gmail settings
- Configure Uptrace:
mailer:
smtp:
enabled: true
host: smtp.gmail.com
port: 587
username: youraccount@gmail.com
password: your_app_specific_password # Not your regular password
from: 'youraccount@gmail.com'
Telegram Integration
Enable Telegram notifications for alerts:
##
## Telegram bot configuration
## Setup guide: https://core.telegram.org/bots#6-botfather
##
telegram:
# Bot token from @BotFather
bot_token: 'your_telegram_bot_token'
System Configuration
Application Logging
Control Uptrace's internal logging:
##
## Application logging configuration
##
logging:
# Log levels: DEBUG, INFO, WARN, ERROR
# DEBUG: Verbose output for troubleshooting
# INFO: Standard operational information
# WARN: Warnings and errors only
# ERROR: Errors only
level: INFO
Premium Features
Enable advanced features with a license:
##
## Premium Edition license
## Enables: distributed tables, advanced SSO, priority support
##
license:
key: 'your_premium_license_key'
Premium Features Include:
- ClickHouse distributed tables for horizontal scaling
- Advanced SSO integrations
- Priority technical support
- Enhanced security features
Deployment Patterns
Production Checklist
Security Requirements:
- ✅ Change all default passwords and tokens in
seed_data
- ✅ Configure TLS/SSL certificates for all connections
- ✅ Set up authentication restrictions (
auth.email_regexp
) - ✅ Enable secure database connections
- ✅ Configure proper firewall rules
High Availability Setup:
- ✅ Deploy multiple Uptrace instances behind load balancer
- ✅ Configure PostgreSQL with read replicas
- ✅ Set up ClickHouse replication (minimum 3 nodes)
- ✅ Configure Redis for caching and session storage
- ✅ Set up database backup procedures
Performance Optimization:
- ✅ Tune telemetry processing settings based on volume
- ✅ Configure appropriate resource limits
- ✅ Set up monitoring for Uptrace itself
- ✅ Configure data retention policies
- ✅ Optimize ClickHouse storage policies
Operational Requirements:
- ✅ Configure email notifications for alerts
- ✅ Set up log aggregation for Uptrace logs
- ✅ Document recovery procedures
- ✅ Configure backup verification
- ✅ Set up monitoring dashboards
Reverse Proxy Configuration
When deploying behind a reverse proxy, ensure proper URL configuration:
site:
# Main deployment
url: https://uptrace.mycompany.com
# Subpath deployment
url: https://mycompany.com/uptrace
Load Balancer Health Checks:
- HTTP endpoint:
GET /api/health
- Expected response:
200 OK
Scaling Guidelines
Horizontal Scaling:
- Uptrace instances are stateless and can be scaled horizontally
- Use load balancer to distribute traffic across instances
- Each instance should connect to the same PostgreSQL and ClickHouse databases
Vertical Scaling Recommendations: According to ClickHouse documentation, prefer vertical scaling over horizontal scaling:
Successful deployments with ClickHouse use servers with hundreds of cores, terabytes of RAM, and petabytes of disk space. Scaling vertically first provides cost efficiency, lower operational overhead, and better query performance.
Resource Requirements:
- Uptrace instances: 2-4 CPU cores, 4-8GB RAM per instance
- PostgreSQL: 2-4 CPU cores, 4-8GB RAM, SSD storage
- ClickHouse: Scale based on data volume (prefer large instances)
- Redis: 1-2 CPU cores, 2-4GB RAM
Troubleshooting Guide
Common Configuration Issues
Database Connection Problems:
# Check connection settings
pg:
addr: localhost:5432 # Verify host and port
user: uptrace # Ensure user exists
password: uptrace # Verify password
database: uptrace # Ensure database exists
TLS Certificate Issues:
# Verify certificate validity
openssl x509 -in uptrace.crt -text -noout
# Check certificate expiration
openssl x509 -in uptrace.crt -checkend 86400
# Test TLS connection
openssl s_client -connect uptrace.local:443
Performance Problems:
- Monitor ClickHouse resource usage
- Check for dropped telemetry data in metrics
- Review query performance in ClickHouse logs
- Verify network latency between components
Authentication Issues:
- Verify email regex patterns if configured
- Check SSO provider configuration
- Confirm user permissions and project access
- Review authentication logs for error details
Monitoring Uptrace Health
Key Metrics to Monitor:
- Database connection health
- Telemetry data ingestion rates
- Query response times
- Memory and CPU usage
- Disk space utilization
Health Check Endpoints:
- API health:
GET /api/health
- Database connectivity: Monitor connection metrics
- Redis connectivity: Monitor cache hit rates