Troubleshooting

Uptrace version

Before trying anything else, make sure you are running the latest Uptrace version:

uptrace version

Compare with the latest available version at GitHub Releases.

Uptrace logs

View live Uptrace logs:

sudo journalctl -u uptrace -f

Check the service status:

sudo systemctl status uptrace

By default, Uptrace only logs failed HTTP requests and failed database queries. Use environment
variables to enable verbose logging:

# Log all incoming HTTP requests
HTTPDEBUG=2 uptrace serve

# Log all ClickHouse queries
CHDEBUG=2 uptrace serve

# Log all PostgreSQL queries
PGDEBUG=2 uptrace serve

Database logs

View ClickHouse logs:

sudo tail -f /var/log/clickhouse-server/clickhouse-server.err.log -n 100

View PostgreSQL logs:

sudo tail -f /var/log/postgresql/postgresql-17-main.log

Resetting ClickHouse database

If ClickHouse queries are failing, try resetting the database:

uptrace ch reset

To check the database status:

uptrace ch status

Distributed and Kafka tables

In sharded (Premium) and Kafka-based (Enterprise) deployments, uptrace ch migrate already creates the distributed tables, the ClickHouse Kafka engine tables, and the materialized views that connect them. The commands below let you recreate or tear down those objects by hand — useful after adding a shard, changing Kafka engine settings, or pausing ingestion for maintenance. They operate on every read-write replica.

Distributed tables

Recreate the distributed tables (*_dist, plus the uptrace_all variants) on each replica:

uptrace ch dist

This is a no-op unless ch_cluster.distributed is enabled. It is safe to run at any time without holding the migration lock — run it after adding a shard or if a distributed table is missing.

Kafka engine tables

Recreate the ClickHouse Kafka engine tables (<table>_queue) that consume from the Kafka topics. The broker list comes from ch_cluster.kafka.brokers, and the number of consumers is sized from the topic's partition count:

uptrace ch kafka_create

Pass --setting (repeatable) to override or add Kafka engine settings on the created tables:

uptrace ch kafka_create --setting "kafka_max_block_size = 1048576"

The tables are created with CREATE OR REPLACE, so it is safe to re-run.

Kafka materialized views

The materialized views (<table>_queue_mv) are what actually move rows from each Kafka queue table into its destination table, so creating them starts ingestion and dropping them stops it without touching the queue tables or the data buffered in Kafka.

Create (or recreate) the views:

uptrace ch kafka_create_view

It also accepts repeatable --setting flags.

Drop the views to pause consumption from Kafka — for example before a maintenance window:

uptrace ch kafka_drop_view

Recreate them with uptrace ch kafka_create_view to resume. While the views are dropped, data accumulates in Kafka and is consumed once they are recreated.

Resetting PostgreSQL database

You can also reset the PostgreSQL database that Uptrace uses to store metadata:

uptrace pg reset

To check the database status:

uptrace pg status

Re-seeding the database

After resetting ClickHouse or PostgreSQL, you may need to re-seed the database with default data:

uptrace db seed

TLS certificate issues

Verify that the certificate is valid and not expired:

openssl x509 -in uptrace.crt -text -noout
openssl x509 -in uptrace.crt -checkend 86400

Test the TLS connection:

openssl s_client -connect uptrace.local:443

Health check

Verify that Uptrace is running and can reach its databases:

curl -f http://localhost:14318/api/health