Instrumenting Go database/sql with OpenTelemetry [otelsql]

OpenTelemetry database SQL provides libraries and tools for instrumenting SQL databases such as PostgreSQL, MySQL, and Microsoft SQL Server to collect telemetry data.

database/sql package provides a lightweight and idiomatic interface to row-oriented databases in Golang. It supports most popular relational DBMS via 3-rd party drivers.

This article will teach you how to monitor database/sql performance using OpenTelemetry observability framework.

Quick Reference

There are two main OpenTelemetry instrumentation libraries for Go database/sql:

Both libraries provide automatic query tracing, context propagation, and semantic conventions support. This guide uses uptrace/otelsql, but the concepts apply to both.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that aims to standardize and simplify the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry supports multiple programming languages and platforms, making it suitable for a wide range of applications and environments.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization. When deploying on Kubernetes, the OpenTelemetry Operator streamlines the setup and configuration of OpenTelemetry instrumentation across your cluster. Pair this backend monitoring with the OpenTelemetry Kubernetes guide to observe database workloads alongside cluster health.

OpenTelemetry database/sql

To install otelsql instrumentation:

go get github.com/uptrace/opentelemetry-go-extra/otelsql

• Example
• Reference

Supported Databases

OpenTelemetry database/sql instrumentation works with any Go database driver that implements the standard database/sql interface:

PostgreSQL:

• github.com/lib/pq - Pure Go PostgreSQL driver
• github.com/jackc/pgx - PostgreSQL driver with pgx support

MySQL:

• github.com/go-sql-driver/mysql - MySQL driver

SQLite:

• github.com/mattn/go-sqlite3 - SQLite3 driver (requires CGO)
• modernc.org/sqlite - Pure Go SQLite driver

Microsoft SQL Server:

• github.com/denisenkom/go-mssqldb - SQL Server driver

Others:
Oracle, CockroachDB, TiDB, and any other database with a database/sql compatible driver.

For database-level monitoring (server metrics, replication status, query performance at the database level), you can use OpenTelemetry Collector receivers:

• PostgreSQL monitoring - Monitor PostgreSQL server with Collector
• MySQL monitoring - Monitor MySQL server with Collector

Using database/sql

OpenTelemetry SQL allows developers to monitor the performance and behavior of their SQL databases, providing insights into database usage patterns, resource utilization, and performance bottlenecks

To instrument database/sql, you need to connect to a database using the API provided by otelsql:

import (
    "github.com/uptrace/opentelemetry-go-extra/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

db, err := otelsql.Open("sqlite", "file::memory:?cache=shared",
    otelsql.WithAttributes(semconv.DBSystemSqlite),
    otelsql.WithDBName("mydb"))
if err != nil {
    panic(err)
}

// db is *sql.DB

And then use context-aware API to propagate the active span via context:

var num int
if err := db.QueryRowContext(ctx, "SELECT 42").Scan(&num); err != nil {
    panic(err)
}

Query Tracing Options

By default, otelsql traces queries without including query parameters. You can control this behavior for security or debugging purposes.

Trace queries without parameters (default):

db, err := otelsql.Open("postgres", dsn,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
)
// Query appears in traces as: SELECT * FROM users WHERE id = $1

Trace queries with parameters:

db, err := otelsql.Open("postgres", dsn,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
    otelsql.WithSpanOptions(otelsql.TraceQueryWithArgs()),
)
// Query appears in traces as: SELECT * FROM users WHERE id = 123

Including query arguments can help with debugging, but be cautious about logging sensitive data like passwords or personal information.

Monitoring Connection Pools

Connection pool monitoring helps you identify connection leaks, pool exhaustion, and other resource issues. The otelsql library can automatically report sql.DBStats as OpenTelemetry metrics.

import (
    "github.com/uptrace/opentelemetry-go-extra/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.4.0"
)

db, err := otelsql.Open("postgres", dsn,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
    otelsql.WithDBName("production"),
)
if err != nil {
    panic(err)
}

// Register DBStats metrics
err = otelsql.ReportDBStatsMetrics(db,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
)
if err != nil {
    panic(err)
}

This reports metrics like:

• db.sql.connections.open - Number of established connections
• db.sql.connections.idle - Number of idle connections
• db.sql.connections.wait_count - Total number of connections waited for
• db.sql.connections.wait_duration - Total time blocked waiting for connections

These metrics help you tune connection pool settings:

db.SetMaxOpenConns(25)           // Maximum open connections
db.SetMaxIdleConns(5)            // Maximum idle connections
db.SetConnMaxLifetime(time.Hour) // Maximum connection lifetime

Finding Slow Queries

With instrumentation enabled, all database queries are automatically traced. You can view query durations and identify slow queries in your OpenTelemetry backend.

ctx := context.Background()

// This query is automatically traced with duration
rows, err := db.QueryContext(ctx, `
    SELECT id, email, created_at
    FROM users
    WHERE created_at > $1
    ORDER BY created_at DESC
    LIMIT 100
`, time.Now().AddDate(0, -1, 0))
if err != nil {
    return err
}
defer rows.Close()

Each query creates a span with attributes:

• db.system - Database type (postgresql, mysql, sqlite)
• db.name - Database name
• db.statement - SQL query text
• db.operation - Operation type (SELECT, INSERT, UPDATE, DELETE)

Troubleshooting

Queries not appearing in traces:

Make sure you're using context-aware methods:

// Wrong - no tracing
rows, err := db.Query("SELECT * FROM users")

// Correct - traces enabled
rows, err := db.QueryContext(ctx, "SELECT * FROM users")

Context doesn't contain active span:

Ensure your context has an active span from your HTTP handler or other instrumentation:

// In HTTP handler with OpenTelemetry HTTP instrumentation
func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context() // Contains trace context
    rows, err := db.QueryContext(ctx, "SELECT ...") // Will be traced
}

High memory usage:

Check your connection pool settings and avoid unbounded attributes:

// Limit concurrent connections
db.SetMaxOpenConns(25)
db.SetConnMaxIdleTime(5 * time.Minute)

// Avoid high-cardinality attributes
// Don't: otelsql.WithAttributes(attribute.String("user_id", userId))
// Do: otelsql.WithAttributes(semconv.DBSystemPostgreSQL)

What is Uptrace?

Uptrace is a OpenTelemetry APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues. For Go instrumentation, see the OpenTelemetry Go guide and compare with top APM tools.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

FAQ

What's the difference between otelsql and database-specific monitoring? otelsql instruments your application's database queries at the code level, showing you which queries your application executes and their performance. Database-specific monitoring (like PostgreSQL receiver) monitors the database server itself - CPU usage, disk I/O, replication status, and server-level metrics.

Which databases work with otelsql? Any database with a Go database/sql driver: PostgreSQL, MySQL, SQLite, SQL Server, Oracle, CockroachDB, TiDB, and others. The instrumentation is driver-agnostic.

Does otelsql impact performance? The overhead is minimal - typically 1-5% for most workloads. Tracing is asynchronous and doesn't block query execution.

Can I use otelsql with GORM or other ORMs? Yes, but it's better to use ORM-specific instrumentation for better integration. For GORM, use otelgorm which provides integration with GORM's hooks. For Ent, use Ent instrumentation.

How do I trace queries without exposing sensitive data? By default, otelsql doesn't include query parameters in traces. Use otelsql.TraceQueryWithoutArgs() (the default) to exclude parameters. Only enable otelsql.TraceQueryWithArgs() in development environments.

What's next?

With database/sql instrumentation enabled, you can track query performance, identify slow queries, monitor connection pool health, and optimize database interactions. You can combine this application-level instrumentation with database-level monitoring for complete visibility.

For ORM-level monitoring with additional features like hook tracing and relationship loading visibility, explore GORM instrumentation or Ent instrumentation for more advanced database operations.