Debugging Uptrace issues

Uptrace version

Before trying anything else, make sure you have the latest Uptrace version.

To see the installed Uptrace version:

uptrace version

You can check the latest available version at GitHub Releasesopen in new window.

Uptrace config

You can view the config Uptrace uses with the following command:

uptrace config dump

Logging

You can view Uptrace logs using journalctl command:

sudo journalctl -u uptrace -f

To check the status of the service:

sudo systemctl status uptrace

By default, Uptrace only logs failed HTTP requests and failed ClickHouse queries. You can configure Uptrace to log all incoming HTTP requests with an env variable:

HTTPDEBUG=2 uptrace serve

To log all ClickHouse queries:

CHDEBUG=2 uptrace serve

To log all PostgreSQL queries:

PGDEBUG=2 uptrace serve

Resetting ClickHouse database

If ClickHouse queries are failing, you can try to to reset ClickHouse database:

uptrace ch reset

The check the database status:

uptrace ch status

Resetting PostgreSQL database

Just like with ClickHouse, you can reset the PostgreSQL database that Uptrace uses to store metadata:

uptrace pg reset

The check the database status:

uptrace pg status

Kafka queues

Uptrace Enterprise Edition uses Kafka to queue incoming data for asynchronous processing.

Use the following command to check the number of unprocessed messages in the queues (the "LAG" column):

/usr/local/kafka/bin/kafka-consumer-groups.sh --describe --group kafkaq --bootstrap-server 127.0.0.1:9092

GROUP           TOPIC                    PARTITION  CURRENT-OFFSET  LOG-END-OFFSET  LAG             CONSUMER-ID                              HOST            CLIENT-ID
kafkaq          prod.v1.spans            0          76450884876     76450904337     19461           kgo-abf6bc0b-489c-45bc-ac67-80a6f05c2fdb /10.0.1.1       kgo
kafkaq          prod.v1.datapoints       0          50385576786     50385587980     11194           kgo-f659dd2f-9c65-4408-95c6-9107178ce362 /10.0.1.1       kgo

Last Updated: 7/25/2024, 12:36:08 PM