Uptrace Configuration Guide
Uptrace is an open-source APM and observability platform that supports distributed tracing, metrics, and log management. You can customize all Uptrace settings and modify the generated ClickHouse database schema using a single YAML configuration file.
Configuration File Overview
All Uptrace configuration is managed through a single YAML file. You can create the default configuration file using the Uptrace binary:
./uptrace -config=/path/to/config.yml config create
Configuration File Location
You can specify the configuration file location in two ways:
Using a CLI argument:
uptrace --config=/etc/uptrace/config.yml serve
Using an environment variable:
UPTRACE_CONFIG=/etc/uptrace/config.yml uptrace serve
If you don't explicitly specify the configuration location, Uptrace defaults to /etc/uptrace/config.yml
.
Database Configuration
Uptrace requires two databases: PostgreSQL for metadata storage and ClickHouse for telemetry data.
PostgreSQL Configuration
Uptrace requires a PostgreSQL database to store metadata such as metric names, alerts, and user information. This database is typically small, occupying only a few megabytes of disk space.
Configure PostgreSQL database credentials in your config file:
##
## PostgreSQL database configuration
##
pg:
addr: localhost:5432
user: uptrace
password: uptrace
database: uptrace
# TLS configuration (uncomment to enable)
#tls:
# insecure_skip_verify: true # only for self-signed certificates
High Availability Setup:
For high availability, you can configure PostgreSQL standby nodes:
pg:
addr: primary-host:5432
user: uptrace
password: uptrace
database: uptrace
# Standby replicas for read operations
replicas:
- standby-host1:5432
- standby-host2:5432
See the TLS Client Configuration section for SSL certificate configuration.
ClickHouse Configuration
ClickHouse is the primary storage engine for telemetry data including spans, logs, and metrics. Configure ClickHouse database credentials as follows:
ch_cluster:
cluster: 'uptrace1'
# Enable ClickHouse replication for high availability
# Cluster name is required when replication is enabled
replicated: false
# Enable ClickHouse distributed tables for horizontal scaling
# Requires Premium Edition
distributed: false
shards:
- replicas:
- addr: localhost:9000
user: default
password: ''
database: uptrace
High Availability Setup:
For high availability, configure multiple ClickHouse replicas:
ch_cluster:
cluster: 'uptrace1'
replicated: true
shards:
- replicas:
- addr: replica1:9000
user: default
password: ''
database: uptrace
- addr: replica2:9000
user: default
password: ''
database: uptrace
- addr: replica3:9000
user: default
password: ''
database: uptrace
Environment Variables
You can use environment variables in your YAML configuration file to keep sensitive information secure:
pg:
addr: ${UPTRACE_PG_ADDR:localhost:5432}
user: ${UPTRACE_PG_USER:uptrace}
password: ${UPTRACE_PG_PASSWORD}
database: ${UPTRACE_PG_DATABASE:uptrace}
Environment Variable Format:
- Simple substitution:
${ENV_VAR_NAME}
- With default values:
${ENV_VAR_NAME:default_value}
- To escape
$
: Use$$
Variables are expanded before parsing the YAML content using Go's os.Expand function.
ClickHouse Advanced Configuration
Query Settings
To modify ClickHouse query settings:
ch_cluster:
shards:
- replicas:
- addr: localhost:9000
query_settings:
async_insert: 1 # enabled by default
wait_for_async_insert: 1 # disabled by default
TLS Configuration
To use TLS connections with ClickHouse, you need to enable the secure TCP port (9440) and configure SSL certificates in your ClickHouse configuration:
ClickHouse Server Configuration:
<?xml version="1.0" ?>
<clickhouse>
<tcp_port_secure>9440</tcp_port_secure>
<openSSL>
<server>
<certificateFile>/etc/clickhouse/certs/chnode.crt</certificateFile>
<privateKeyFile>/etc/clickhouse/certs/chnode.key</privateKeyFile>
<verificationMode>relaxed</verificationMode>
<caConfig>/etc/clickhouse/certs/Uptrace_CA.crt</caConfig>
<cacheSessions>true</cacheSessions>
<disableProtocols>sslv2,sslv3</disableProtocols>
<preferServerCiphers>true</preferServerCiphers>
</server>
</openSSL>
</clickhouse>
Uptrace TLS Configuration:
ch_cluster:
shards:
- replicas:
- addr: localhost:9440
user: default
database: uptrace
tls:
ca_file: /etc/clickhouse/certs/Uptrace_CA.crt
cert_file: /etc/clickhouse/certs/chnode.crt
key_file: /etc/clickhouse/certs/chnode.key
insecure_skip_verify: true
ClickHouse Cloud Configuration:
For ClickHouse Cloud, you may need to override the server name if the address doesn't match the certificate:
ch_cluster:
shards:
- replicas:
- addr: tm849a32za.us-central1.gcp.clickhouse.cloud:9440
tls:
server_name_override: 'tm849a32za.us-central1.gcp.clickhouse.cloud'
Performance Considerations
Uptrace performance is primarily limited by ClickHouse database performance. When scaling ClickHouse, prefer vertical scaling over horizontal scaling.
According to the ClickHouse blog:
We commonly find successful deployments with ClickHouse deployed on servers with hundreds of cores, terabytes of RAM, and petabytes of disk space.
Scaling vertically first has a number of benefits, principally cost efficiency, lower cost of ownership (with respect to operations), and better query performance due to the minimization of data on the network for operations such as JOINs.
ClickHouse Schema Configuration
The following options allow you to modify the ClickHouse schema generated by Uptrace.
⚠️ Important: For changes to take effect, you must reset the ClickHouse database:
uptrace ch reset
Compression Configuration
Configure the compression method for ClickHouse tables:
ch_schema:
# Compression codec options: LZ4, ZSTD(1), or Default
compression: ZSTD(1)
# Storage policies for different data types
spans_index:
storage_policy: default
spans_data:
storage_policy: default
logs_index:
storage_policy: default
logs_data:
storage_policy: default
metrics:
storage_policy: default
Replication Setup
To enable ClickHouse table replication, follow these steps:
Step 1: Configure ClickHouse Cluster
Configure a ClickHouse cluster with at least 3 replicas. Important: Keep internal_replication = true
to avoid data duplicates.
<clickhouse>
<remote_servers>
<uptrace1>
<shard>
<internal_replication>true</internal_replication>
<replica>
<host>clickhouse-1</host>
<port>9000</port>
</replica>
<replica>
<host>clickhouse-2</host>
<port>9000</port>
</replica>
<replica>
<host>clickhouse-3</host>
<port>9000</port>
</replica>
</shard>
</uptrace1>
</remote_servers>
</clickhouse>
Step 2: Update Uptrace Configuration
Update your Uptrace configuration to use the cluster name:
ch_cluster:
cluster: 'uptrace1'
# Enable ClickHouse replication
replicated: true
shards:
- replicas:
- addr: clickhouse-1:9000
- addr: clickhouse-2:9000
- addr: clickhouse-3:9000
Step 3: Reset ClickHouse Database
Apply the changes by resetting the ClickHouse database:
uptrace ch reset
Step 4: Verify Replication
Verify replication is working using clickhouse-client
:
SELECT
database,
table,
is_leader,
replica_is_active
FROM system.replicas;
Expected output:
┌─database─┬─table───────────┬─is_leader─┬─replica_is_active────────────────────────┐
│ uptrace │ spans_data │ 1 │ {'replica1':1,'replica2':1,'replica3':1} │
│ uptrace │ spans_index │ 1 │ {'replica1':1,'replica2':1,'replica3':1} │
└──────────┴─────────────────┴───────────┴──────────────────────────────────────────┘
Network Configuration
Port Configuration
By default, Uptrace listens on the following ports:
- Port
4317
for OTLP/gRPC - Port
80
for OTLP/HTTP and web UI
You can customize these ports in your configuration:
listen:
# HTTP server for OTLP/HTTP API and Vue.js UI
http:
addr: ':80'
# gRPC server for OTLP/gRPC API
grpc:
addr: ':4317'
# TLS configuration for both HTTP and gRPC (uncomment to enable)
#tls:
# cert_file: /etc/uptrace/uptrace.crt
# key_file: /etc/uptrace/uptrace.key
Domain Configuration
Configure the Uptrace domain for the Vue-powered UI and DSN/endpoints:
site:
# Public URL for the UI and DSN endpoint
# Port should match listen.http.addr unless using a reverse proxy
url: 'https://uptrace.mydomain.com'
After making changes, restart Uptrace:
sudo systemctl restart uptrace
TLS/SSL Configuration
Uptrace supports multiple TLS configuration options for secure connections.
Automatic TLS with Let's Encrypt
For HTTP and gRPC listeners, Uptrace supports automatic Let's Encrypt certificates using the certmagic library:
##
## Automatic TLS certificate issuance and renewal using Let's Encrypt
##
certmagic:
# Enable Let's Encrypt certificates
enabled: true
# Use Let's Encrypt staging environment (for testing)
staging_ca: false
# Address for HTTP-01 challenge validation
http_challenge_addr: ':80'
Manual TLS Server Configuration
For custom certificates, first generate a certificate (replace localhost
with your domain):
openssl req -x509 -newkey rsa:4096 -sha256 -days 3650 -nodes \
-keyout uptrace.key -out uptrace.crt -subj "/CN=localhost" \
-addext "subjectAltName=DNS:localhost"
Add the certificate to trusted certificates:
sudo cp uptrace.crt /usr/local/share/ca-certificates/
sudo update-ca-certificates
Configure Uptrace to use the certificate:
listen:
grpc:
addr: ':4317'
http:
addr: ':443'
tls:
cert_file: config/tls/uptrace.crt
key_file: config/tls/uptrace.key
#ca_file: path/to/ca_file
site:
# Update domain for Vue-powered UI
url: 'https://uptrace.mydomain.com'
Complete TLS server options:
listen:
tls:
# Path to CA certificate (optional)
ca_file: ''
# Path to TLS certificate
cert_file: ''
# Path to TLS private key
key_file: ''
# Minimum TLS version (default: TLS 1.2)
min_version: ''
# Maximum TLS version (optional)
max_version: ''
# Client certificate verification
client_ca_file: ''
TLS Client Configuration
For database connections, you can configure TLS in several ways:
Use system certificates:
pg:
tls: {}
Skip certificate verification (insecure):
pg:
tls:
insecure_skip_verify: true
Use custom certificates:
pg:
tls:
cert_file: path/to/client.crt
key_file: path/to/client.key
ca_file: path/to/ca.crt
Disable TLS:
pg:
tls:
insecure: true
Override server name:
ch_cluster:
shards:
- replicas:
- addr: tm849a32za.us-central1.gcp.clickhouse.cloud:9440
tls:
server_name_override: 'tm849a32za.us-central1.gcp.clickhouse.cloud'
Complete TLS client options:
tls:
# Disable client transport security (default: false)
insecure: false
# Path to CA certificate (optional)
ca_file: ''
# Path to client certificate (optional)
cert_file: ''
# Path to client private key (optional)
key_file: ''
# Minimum TLS version (default: TLS 1.2)
min_version: ''
# Maximum TLS version (optional)
max_version: ''
# Skip certificate verification (default: false)
insecure_skip_verify: false
# Override server name (optional)
server_name_override: ''
User Authentication and Initial Setup
Initial Data Configuration
On first startup, Uptrace creates a default user with login admin@uptrace.local
and password admin
. You can customize this initial data in the seed_data
section:
# Initial data created on first startup
seed_data:
users:
- name: Admin
key: user1 # Unique identifier for referencing in relationships
email: admin@uptrace.local
password: admin # TODO: Change default password in production!
auth_tokens: [user1_secret]
orgs:
- name: Default Organization
key: org1 # Unique identifier for referencing below
org_users:
- org: org1 # References org.key above
user: user1 # References user.key above
role: owner
projects:
- name: Default Project
key: project1 # Unique identifier for referencing below
org: org1 # References org.key above
tokens: [project1_secret]
project_users:
- project: project1 # References project.key above
user: user1 # References user.key above
perm_level: admin
Authentication Configuration
Control user registration and authentication:
auth:
# Set to true to disable username/password authentication
# Note: SSO authentication will remain available
disabled: false
# Email address validation pattern (optional)
# Only users with matching email addresses can register/login
# Example: '^.+@example\.com$' restricts to @example.com domain
email_regexp: '^.+@example\.com$'
Single Sign-On (SSO) Integration
Uptrace supports integration with popular identity providers:
Email Notifications
To enable email notifications for alerts, configure an SMTP mailer:
##
## SMTP configuration for email notifications
## Documentation: https://uptrace.dev/features/alerting
##
mailer:
smtp:
# Enable this mailer for sending emails
enabled: true
# SMTP server host
host: localhost
# SMTP server port
port: 1025
# Authentication username
username: mailhog
# Authentication password
password: mailhog
# Sender email address
from: 'uptrace@localhost'
# TLS configuration (uncomment to disable opportunistic TLS)
#tls:
# disabled: true
Gmail Configuration
Gmail requires an app-specific password instead of your regular password:
- In Gmail, click your avatar → "Manage your Google Account"
- Navigate to "Security" in the left sidebar
- Scroll to "Signing in to Google" and click "App password"
- Generate a new app password for Uptrace
See Gmail's documentation for detailed instructions.
Gmail SMTP Configuration:
mailer:
smtp:
enabled: true
host: smtp.gmail.com
port: 587
username: your-email@gmail.com
password: your-app-password
from: 'your-email@gmail.com'
Caching Configuration
Uptrace uses Redis for caching to improve query performance:
redis_cache:
addrs:
1: localhost:6379
# Redis authentication
username: ''
password: ''
db: 0
# TLS configuration (uncomment for secure connections)
#tls:
# insecure_skip_verify: true
Reverse Proxy Configuration
When running Uptrace behind a reverse proxy (Nginx, HAProxy, etc.), configure the domain name so Uptrace can render links and redirects correctly:
site:
url: 'https://uptrace.mydomain.com'
Subpath deployment (e.g., http://mydomain.com/uptrace
):
site:
url: 'https://mydomain.com/uptrace'
Scaling and High Availability
Uptrace is stateless and can be scaled horizontally by launching multiple nodes and load balancing traffic.
Default Ports
Service | Port |
---|---|
OTLP HTTP and API | 80 |
OTLP gRPC | 4317 |
High Availability Setup
For high availability:
- Deploy multiple Uptrace instances behind a load balancer
- Configure PostgreSQL with standby replicas for metadata redundancy
- Set up ClickHouse replication for telemetry data redundancy
- Use a load balancer (HAProxy, Nginx) to distribute traffic across instances
Premium Features
License Configuration
For Premium Edition features, specify your Uptrace license key:
license:
key: '<YOUR_LICENSE_KEY>'
Distributed Tables
Distributed tables for horizontal scaling (requires Premium Edition):
ch_cluster:
# Enable ClickHouse distributed tables for horizontal scaling
# Requires Premium Edition
distributed: true
Quick Start Guide
Initial Setup
Follow these steps to get Uptrace running:
- Create configuration file:shell
./uptrace -config=/etc/uptrace/config.yml config create
- Configure databases:
- Set PostgreSQL credentials
- Set ClickHouse credentials
- Configure basic settings:
- Set site URL
- Configure ports (if needed)
- Set up TLS (recommended for production)
- Start Uptrace:shell
uptrace --config=/etc/uptrace/config.yml serve
- Access the UI:
- Default login:
admin@uptrace.local
- Default password:
admin
- Default login:
Production Deployment Checklist
For production deployments, ensure you:
- ✅ Change default credentials in
seed_data
- ✅ Configure TLS/SSL certificates
- ✅ Set up proper database backups
- ✅ Configure email notifications
- ✅ Set up monitoring and alerting
- ✅ Configure data retention policies
- ✅ Enable authentication restrictions if needed
- ✅ Set up high availability (multiple instances, database replication)
- ✅ Configure reverse proxy if using one
Troubleshooting
Common Issues
Database connection errors:
- Verify database credentials and connectivity
- Check firewall rules for database ports
- Ensure database services are running
TLS certificate errors:
- Verify certificate paths and permissions
- Check certificate expiration dates
- Ensure certificate matches the configured domain
Performance issues:
- Monitor ClickHouse performance and resources
- Consider vertical scaling for ClickHouse
- Review query performance and optimization
Authentication problems:
- Check email regexp patterns if configured
- Verify SSO configuration if using external providers
- Ensure proper permissions are set for users and projects