Troubleshooting

Uptrace version

Before trying anything else, make sure you are running the latest Uptrace version:

uptrace version

Compare with the latest available version at GitHub Releases.

Check the deployment

uptrace doctor checks every store and endpoint that your config uses, and reports each one. Run it first: it finds most configuration and connectivity problems in one pass.

uptrace --config=/etc/uptrace/config.yml doctor

ok    build_keys                                the hosted build cannot sign licenses
ok    postgres_primary localhost:5432           PostgreSQL 17.7
ok    postgres_version localhost:5432           PostgreSQL 17.7
skip  postgres_conn_pool                        pg.conn_pool_addr is not set
skip  postgres_replica                          pg.read_replicas is not set
ok    clickhouse_replica shard1/localhost:9000  read-write, replication queue 0, delay 0s
ok    clickhouse_version shard1/localhost:9000  ClickHouse 26.3.35.3
ok    clickhouse_disks shard1/localhost:9000    default (Local) 25% free
ok    clickhouse_fence shard1/localhost:9000    no table is fenced
skip  clickhouse_cluster                        ch_cluster.replicated and ch_cluster.distributed are off
ok    redis                                     1/1 shards answering
skip  kafka_brokers                             kafka.brokers is not set
skip  kafka_consumer_group                      kafka.brokers is not set
warn  migrations                                1 of 34 migrations are not applied; run `uptrace migrate up`
ok    capability_holds                          no capability is held or stopped
ok    runtime_dir                               /run/uptrace is writable, 61% free
skip  certmagic_cache_dir                       certmagic.cache_dir is not used
ok    egress_smtp smtp.example.com:465          1 addresses, TLS 1.3
warn  egress_telegram api.telegram.org:443      TCP: dial tcp 149.154.167.220:443: i/o timeout
doctor: 0 failed, 2 warnings, 12 ok, 5 skipped

The command connects with the same clients, credentials, and TLS settings as Uptrace itself, so a wrong password or a bad certificate fails here, not only an unreachable host. Every check runs, even after a failure, so you see every problem at once.

Each check ends with one status:

uptrace doctor exits with status 1 when a check fails, and 0 otherwise, so you can run it from a script or a cron job. A warning does not change the exit status.

What it checks

Check the running processes

A fresh uptrace doctor process can see a different network than the running Uptrace service, for example inside a container. To get the report of the running processes themselves, add --live:

uptrace doctor --live

--live reads each uptrace-serve.sock and uptrace-worker.sock in service.runtime_dir and prints the report of each process. Run it on the host where Uptrace runs, as a user that can read that directory. In this mode, an unapplied migration is a failure, because the process already serves with a schema older than its version.

The same socket serves GET /status with the version, config, migrations, and readiness of the process. Its memory section shows the soft memory limit of the Go runtime and where it came from (applied from the cgroup limit, env from GOMEMLIMIT, disabled, or no_limit):

curl --unix-socket /run/uptrace/uptrace-serve.sock http://debug/status

JSON output

Add --json to get a report that a script can read:

uptrace doctor --json

{
  "status": "warn",
  "mode": "fresh",
  "checkedAt": "2026-10-01T12:00:00Z",
  "checks": [
    {
      "id": "postgres_primary",
      "target": "localhost:5432",
      "gate": true,
      "status": "ok",
      "detail": "PostgreSQL 17.7",
      "tookMs": 3
    }
  ]
}

status is the most severe status of all checks. With --live --json, the output is a list with one {"socket": ..., "report": ...} entry for each process.

Flags:

Which command to use

Uptrace logs

View live Uptrace logs:

sudo journalctl -u uptrace -f

Check the service status:

sudo systemctl status uptrace

By default, Uptrace only logs failed HTTP requests and failed database queries. Use environment
variables to enable verbose logging:

# Log all incoming HTTP requests
HTTPDEBUG=2 uptrace serve

# Log all ClickHouse queries
CHDEBUG=2 uptrace serve

# Log all PostgreSQL queries
PGDEBUG=2 uptrace serve

Database logs

View ClickHouse logs:

sudo tail -f /var/log/clickhouse-server/clickhouse-server.err.log -n 100

View PostgreSQL logs:

sudo tail -f /var/log/postgresql/postgresql-17-main.log

Resetting ClickHouse database

If ClickHouse queries are failing, try resetting the database. This deletes all telemetry data:

uptrace migrate reset --database ch
uptrace migrate up

To check the migration status of both databases:

uptrace migrate status

Distributed and Kafka tables

In sharded (Premium) and Kafka-based (Enterprise) deployments, uptrace migrate up already creates the distributed tables, the ClickHouse Kafka engine tables, and the materialized views that connect them. The commands below let you recreate or tear down those objects by hand — useful after adding a shard, changing Kafka engine settings, or pausing ingestion for maintenance. They operate on every read-write replica.

Distributed tables

Recreate the distributed tables (*_dist, plus the uptrace_all variants) on each replica:

uptrace ch dist

This is a no-op unless ch_cluster.distributed is enabled. It is safe to run at any time without holding the migration lock — run it after adding a shard or if a distributed table is missing.

Kafka engine tables

Recreate the ClickHouse Kafka engine tables (<table>_queue) that consume from the Kafka topics. The broker list comes from ch_cluster.kafka.brokers, and the number of consumers is sized from the topic's partition count:

uptrace ch kafka_create

Pass --setting (repeatable) to override or add Kafka engine settings on the created tables:

uptrace ch kafka_create --setting "kafka_max_block_size = 1048576"

The tables are created with CREATE OR REPLACE, so it is safe to re-run.

Kafka materialized views

The materialized views (<table>_queue_mv) are what actually move rows from each Kafka queue table into its destination table, so creating them starts ingestion and dropping them stops it without touching the queue tables or the data buffered in Kafka.

Create (or recreate) the views:

uptrace ch kafka_create_view

It also accepts repeatable --setting flags.

Drop the views to pause consumption from Kafka — for example before a maintenance window:

uptrace ch kafka_drop_view

Recreate them with uptrace ch kafka_create_view to resume. While the views are dropped, data accumulates in Kafka and is consumed once they are recreated.

Resetting PostgreSQL database

You can also reset the PostgreSQL database that Uptrace uses to store metadata:

uptrace migrate reset --database pg
uptrace migrate up
uptrace db seed

To check the migration status:

uptrace migrate status

Re-seeding the database

After resetting ClickHouse or PostgreSQL, you may need to re-seed the database with default data:

uptrace db seed

TLS certificate issues

Verify that the certificate is valid and not expired:

openssl x509 -in uptrace.crt -text -noout
openssl x509 -in uptrace.crt -checkend 86400

Test the TLS connection:

openssl s_client -connect uptrace.local:443

Health endpoints

Uptrace serves two health endpoints on its HTTP port. They answer different questions, so point each probe at the right one:

curl -f http://localhost:14318/health/live
curl -f http://localhost:14318/health/ready

Use /health/live for a liveness probe, and /health/ready for a readiness probe or a load balancer. A liveness probe that reads the databases restarts Uptrace whenever a database is down, which does not fix the database and makes the outage longer. Redis never makes Uptrace unready.

From a stop signal until the process exits, /health/ready returns 503 while /health/live stays 200. Set service.shutdown.delay to keep the servers serving for a while after the stop signal, so a load balancer stops routing to the process before its ports close.

On Kubernetes:

livenessProbe:
  httpGet:
    path: /health/live
    port: 14318
readinessProbe:
  httpGet:
    path: /health/ready
    port: 14318

/health/ready returns only {"ready": true} or {"ready": false}. It does not name the dependency that failed, because anyone who reaches the API can read it. To see which one failed, run uptrace doctor --live on the host.

GET /health/status and GET /ping are older liveness checks. They still work.