Instrumenting Go database/sql with OpenTelemetry [otelsql]

OpenTelemetry database/sql instrumentation traces every SQL query your Go application executes, helping you find slow queries, monitor connection pool health, and understand database performance in production.

Quick Setup

Minimal working example:

import (
    "context"
    "github.com/uptrace/opentelemetry-go-extra/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
    _ "github.com/lib/pq"
)

func main() {
    db, err := otelsql.Open("postgres", "postgres://localhost:5432/mydb?sslmode=disable",
        otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
        otelsql.WithDBName("mydb"),
    )
    if err != nil {
        panic(err)
    }

    var count int
    err = db.QueryRowContext(context.Background(), "SELECT count(*) FROM users").Scan(&count)
}

Every QueryContext, ExecContext, and PrepareContext call is automatically traced with query text, duration, and database attributes.

Library Comparison

There are two main OpenTelemetry instrumentation libraries for Go database/sql:

Both libraries provide automatic query tracing, context propagation, and semantic conventions support. This guide uses uptrace/otelsql, but the concepts apply to both.

What is OpenTelemetry?

OpenTelemetry is an open-source observability framework that standardizes the collection, processing, and export of telemetry data from applications and systems.

OpenTelemetry enables developers to instrument their code and collect telemetry data, which can then be exported to various OpenTelemetry backends or observability platforms for analysis and visualization. The OpenTelemetry architecture provides a modular, vendor-neutral approach to observability.

Installation

To install otelsql instrumentation:

go get github.com/uptrace/opentelemetry-go-extra/otelsql

• Example
• Reference

Supported Databases

OpenTelemetry database/sql instrumentation works with any Go database driver that implements the standard database/sql interface:

PostgreSQL:

• github.com/lib/pq - Pure Go PostgreSQL driver
• github.com/jackc/pgx - PostgreSQL driver with pgx support

MySQL:

• github.com/go-sql-driver/mysql - MySQL driver

SQLite:

• github.com/mattn/go-sqlite3 - SQLite3 driver (requires CGO)
• modernc.org/sqlite - Pure Go SQLite driver

Microsoft SQL Server:

• github.com/denisenkom/go-mssqldb - SQL Server driver

Others:
Oracle, CockroachDB, TiDB, and any other database with a database/sql compatible driver.

For database-level monitoring (server metrics, replication status, query performance at the database level), you can use OpenTelemetry Collector receivers:

• PostgreSQL monitoring - Monitor PostgreSQL server with Collector
• MySQL monitoring - Monitor MySQL server with Collector

Using database/sql

To instrument database/sql, replace standard sql.Open or sql.OpenDB with their otelsql equivalents:

import (
    "github.com/uptrace/opentelemetry-go-extra/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

db, err := otelsql.Open("sqlite", "file::memory:?cache=shared",
    otelsql.WithAttributes(semconv.DBSystemSqlite),
    otelsql.WithDBName("mydb"))
if err != nil {
    panic(err)
}

// db is *sql.DB

And then use context-aware API to propagate the active span via context:

var num int
if err := db.QueryRowContext(ctx, "SELECT 42").Scan(&num); err != nil {
    panic(err)
}

Query Tracing Options

By default, otelsql traces queries without including query parameters. You can control this behavior for security or debugging purposes.

Trace queries without parameters (default):

db, err := otelsql.Open("postgres", dsn,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
)
// Query appears in traces as: SELECT * FROM users WHERE id = $1

Trace queries with parameters:

db, err := otelsql.Open("postgres", dsn,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
    otelsql.WithSpanOptions(otelsql.TraceQueryWithArgs()),
)
// Query appears in traces as: SELECT * FROM users WHERE id = 123

Including query arguments can help with debugging, but be cautious about logging sensitive data like passwords or personal information.

Monitoring Connection Pools

Connection pool monitoring helps you identify connection leaks, pool exhaustion, and other resource issues. The otelsql library can automatically report sql.DBStats as OpenTelemetry metrics.

import (
    "github.com/uptrace/opentelemetry-go-extra/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

db, err := otelsql.Open("postgres", dsn,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
    otelsql.WithDBName("production"),
)
if err != nil {
    panic(err)
}

// Register DBStats metrics
err = otelsql.ReportDBStatsMetrics(db,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
)
if err != nil {
    panic(err)
}

This reports metrics like:

• db.sql.connections.open - Number of established connections
• db.sql.connections.idle - Number of idle connections
• db.sql.connections.wait_count - Total number of connections waited for
• db.sql.connections.wait_duration - Total time blocked waiting for connections

These metrics help you tune connection pool settings:

db.SetMaxOpenConns(25)           // Maximum open connections
db.SetMaxIdleConns(5)            // Maximum idle connections
db.SetConnMaxLifetime(time.Hour) // Maximum connection lifetime

Finding Slow Queries

With instrumentation enabled, all database queries are automatically traced. You can view query durations and identify slow queries in your OpenTelemetry backend.

ctx := context.Background()

// This query is automatically traced with duration
rows, err := db.QueryContext(ctx, `
    SELECT id, email, created_at
    FROM users
    WHERE created_at > $1
    ORDER BY created_at DESC
    LIMIT 100
`, time.Now().AddDate(0, -1, 0))
if err != nil {
    return err
}
defer rows.Close()

Each query creates a span with attributes:

• db.system - Database type (postgresql, mysql, sqlite)
• db.name - Database name
• db.statement - SQL query text
• db.operation - Operation type (SELECT, INSERT, UPDATE, DELETE)

Using with pgx Connector

For PostgreSQL applications using pgx (the most popular Go PostgreSQL driver), you can use otelsql.OpenDB with a pgx connector for better performance and feature support:

import (
    "github.com/jackc/pgx/v5/stdlib"
    "github.com/uptrace/opentelemetry-go-extra/otelsql"
    semconv "go.opentelemetry.io/otel/semconv/v1.26.0"
)

connector, err := stdlib.GetDefaultDriver().OpenConnector("postgres://localhost:5432/mydb")
if err != nil {
    panic(err)
}

db := otelsql.OpenDB(connector,
    otelsql.WithAttributes(semconv.DBSystemPostgreSQL),
    otelsql.WithDBName("mydb"),
)

Framework Integration

otelsql works with any Go web framework. The key is passing the request context to database calls so spans are linked to the parent HTTP trace.

With Gin:

func getUsers(c *gin.Context) {
    ctx := c.Request.Context() // Contains trace from otelgin middleware
    rows, err := db.QueryContext(ctx, "SELECT id, name FROM users")
    // ...
}

With Echo:

func getUsers(c echo.Context) error {
    ctx := c.Request().Context() // Contains trace from otelecho middleware
    rows, err := db.QueryContext(ctx, "SELECT id, name FROM users")
    // ...
}

With net/http:

func getUsers(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context() // Contains trace from otelhttp middleware
    rows, err := db.QueryContext(ctx, "SELECT id, name FROM users")
    // ...
}

Troubleshooting

Queries not appearing in traces:

Make sure you're using context-aware methods. This is the most common mistake:

// Wrong - no tracing (context not passed)
rows, err := db.Query("SELECT * FROM users")

// Correct - traces enabled
rows, err := db.QueryContext(ctx, "SELECT * FROM users")

The same applies to all database operations:

// Wrong                              // Correct
db.Exec(query)                        db.ExecContext(ctx, query)
db.QueryRow(query)                    db.QueryRowContext(ctx, query)
db.Prepare(query)                     db.PrepareContext(ctx, query)
tx.Commit()                           tx.Commit() // transactions auto-traced

Context doesn't contain active span:

Ensure your context has an active span from your HTTP handler or other instrumentation:

func handler(w http.ResponseWriter, r *http.Request) {
    ctx := r.Context() // Contains trace context from HTTP middleware
    rows, err := db.QueryContext(ctx, "SELECT ...") // Will be traced as child span
}

If you need to create a root span for background jobs or CLI tools:

ctx, span := tracer.Start(context.Background(), "batch-job")
defer span.End()

rows, err := db.QueryContext(ctx, "SELECT ...")

High memory usage:

Check your connection pool settings and avoid unbounded attributes:

// Limit concurrent connections
db.SetMaxOpenConns(25)
db.SetMaxIdleConns(5)
db.SetConnMaxIdleTime(5 * time.Minute)
db.SetConnMaxLifetime(time.Hour)

// Avoid high-cardinality attributes
// Don't: otelsql.WithAttributes(attribute.String("user_id", userId))
// Do: otelsql.WithAttributes(semconv.DBSystemPostgreSQL)

Duplicate or missing spans in distributed traces:

Ensure you're propagating context correctly through goroutines:

// Wrong - background goroutine loses trace context
go func() {
    db.QueryContext(context.Background(), "SELECT ...")
}()

// Correct - pass the parent context
go func(ctx context.Context) {
    db.QueryContext(ctx, "SELECT ...")
}(ctx)

What is Uptrace?

Uptrace is an OpenTelemetry APM that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demo (no login required) or running it locally with Docker. The source code is available on GitHub.

FAQ

What's the difference between otelsql and database-specific monitoring? otelsql instruments your application's database queries at the code level, showing you which queries your application executes and their performance. Database-specific monitoring (like the PostgreSQL receiver) monitors the database server itself - CPU usage, disk I/O, replication status, and server-level metrics. Use both for complete visibility.

Which databases work with otelsql? Any database with a Go database/sql driver: PostgreSQL, MySQL, SQLite, SQL Server, Oracle, CockroachDB, TiDB, and others. The instrumentation is driver-agnostic.

Does otelsql impact performance? The overhead is minimal - typically 1-5% for most workloads. Span creation is lightweight and export is asynchronous, so it doesn't block query execution.

Can I use otelsql with GORM or other ORMs? Yes, but it's better to use ORM-specific instrumentation for better integration. For GORM, use otelgorm which provides integration with GORM's hooks. For Ent, use Ent instrumentation.

How do I trace queries without exposing sensitive data? By default, otelsql doesn't include query parameters in traces. Use otelsql.TraceQueryWithoutArgs() (the default) to exclude parameters. Only enable otelsql.TraceQueryWithArgs() in development environments.

Should I use XSAM/otelsql or uptrace/otelsql? Both are well-maintained and functional. XSAM/otelsql has more community adoption; uptrace/otelsql includes built-in DBStats metrics reporting. Choose either based on your preference - both follow the same patterns.

What's next?

With database/sql instrumentation enabled, you can track query performance, identify slow queries, monitor connection pool health, and optimize database interactions.

Next steps to enhance your observability:

• Add ORM-level tracing with GORM instrumentation or Ent instrumentation
• Monitor the database server itself with PostgreSQL or MySQL receivers
• Add HTTP framework tracing with Gin, Echo, or net/http
• Correlate database spans with logs using slog or Zap
• Create custom spans using the OpenTelemetry Go Tracing API