OpenTelemetry Manual Instrumentation in Java: Custom Spans & Business Metrics

November 12, 2025

11 min read

Manual instrumentation with OpenTelemetry gives you complete control over what telemetry data your Java application captures. This guide demonstrates how to create custom spans, record business metrics, and add instrumentation to Spring Boot applications using the OpenTelemetry Java API.

Complete Working Example

Before diving into theory, let's see a complete working example that demonstrates the key concepts. This code shows how to create custom spans with business context, handle errors, and record metrics.

java

package com.example.orders;

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.SpanKind;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.metrics.LongCounter;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.context.Scope;
import org.springframework.stereotype.Service;

@Service
public class OrderProcessingService {

    private final Tracer tracer;
    private final LongCounter ordersProcessed;

    public OrderProcessingService() {
        // Get tracer for creating spans
        this.tracer = GlobalOpenTelemetry.getTracer(
            "com.example.orders",  // Instrumentation scope name
            "1.0.0"                 // Version
        );

        // Get meter for recording metrics
        Meter meter = GlobalOpenTelemetry.getMeter("com.example.orders");
        this.ordersProcessed = meter
            .counterBuilder("orders.processed")
            .setDescription("Total number of orders processed")
            .setUnit("orders")
            .build();
    }

    public OrderResult processOrder(String orderId, double amount) {
        // Create a custom span for this business operation
        Span span = tracer.spanBuilder("process_order")
            .setSpanKind(SpanKind.INTERNAL)
            .setAttribute("order.id", orderId)
            .setAttribute("order.amount", amount)
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            // Your business logic runs here
            // The span is automatically the "current" span

            // Add business event
            span.addEvent("order.validation.started");
            validateOrder(orderId, amount);
            span.addEvent("order.validation.completed");

            // Nested operation creates child span automatically
            PaymentResult payment = processPayment(orderId, amount);
            span.setAttribute("payment.id", payment.getId());
            span.setAttribute("payment.status", payment.getStatus());

            // Record successful processing
            span.setStatus(StatusCode.OK);
            ordersProcessed.add(1);

            return new OrderResult(orderId, "SUCCESS");

        } catch (ValidationException e) {
            // Record business error
            span.setStatus(StatusCode.ERROR, "Validation failed");
            span.recordException(e);
            span.setAttribute("error.type", "validation");
            throw e;

        } catch (Exception e) {
            // Record unexpected error
            span.setStatus(StatusCode.ERROR, "Processing failed");
            span.recordException(e);
            throw new OrderProcessingException("Failed to process order", e);

        } finally {
            // Always end the span
            span.end();
        }
    }

    private PaymentResult processPayment(String orderId, double amount) {
        // Child span created automatically because parent is current
        Span span = tracer.spanBuilder("process_payment")
            .setSpanKind(SpanKind.CLIENT)
            .setAttribute("payment.amount", amount)
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            // Payment processing logic
            // ...
            span.setStatus(StatusCode.OK);
            return new PaymentResult("PAY-" + orderId, "COMPLETED");
        } finally {
            span.end();
        }
    }

    private void validateOrder(String orderId, double amount) {
        if (amount <= 0) {
            throw new ValidationException("Invalid amount: " + amount);
        }
    }
}

What this example demonstrates:

Tracer initialization - Getting a tracer instance from GlobalOpenTelemetry
Custom span creation - Building spans with business-specific names and attributes
Nested spans - Child spans automatically linked to parent when using makeCurrent()
Business events - Recording important milestones within a span
Error handling - Properly capturing exceptions and setting span status
Metrics - Recording business counters alongside traces
Resource cleanup - Using try-with-resources to ensure spans end properly

Time to implement: 15 minutes | Lines of code: ~70 | Complexity: Medium

This pattern covers 80% of manual instrumentation needs. The sections below explain each concept in detail.

Why Manual Instrumentation?

Automatic instrumentation with the OpenTelemetry Java Agent handles framework-level operations like HTTP requests and database queries. However, it cannot capture business-specific operations that are unique to your application.

Manual instrumentation fills this gap by allowing you to:

Capture business operations - Track order processing, payment workflows, report generation, or any domain-specific logic that automatic instrumentation misses.

Add business context - Enrich spans with customer IDs, order amounts, product categories, or other business attributes that help you understand user behavior and system performance.

Create custom metrics - Record business KPIs like orders per minute, cart abandonment rates, or subscription conversions that matter to your business.

Measure critical sections - Time specific code blocks, algorithms, or computations that are important for performance optimization.

Correlate business events - Track the relationship between business actions (e.g., add to cart → checkout → payment → fulfillment) across services.

Core Concepts

Manual instrumentation in OpenTelemetry Java revolves around three main concepts: Tracers, Spans, and Context. Understanding these building blocks is essential for effective instrumentation.

Tracer

A Tracer is the entry point for creating spans. It acts as a factory for span builders and represents your instrumentation scope.

java

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;

// Get a tracer for your service
Tracer tracer = GlobalOpenTelemetry.getTracer(
    "com.example.myservice",  // Scope name (usually package or service name)
    "1.0.0"                    // Version (optional but recommended)
);

Each tracer has a name and optional version that identifies the instrumentation library. This helps filter and organize telemetry data in your observability backend.

Span

A span represents a unit of work in your application. It has a start time, end time, name, and attributes that describe the operation.

java

import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.StatusCode;

// Create and configure a span
Span span = tracer.spanBuilder("calculate_discount")
    .setAttribute("customer.tier", "premium")
    .setAttribute("discount.code", "SAVE20")
    .startSpan();

try {
    // Your code here
    span.setStatus(StatusCode.OK);
} catch (Exception e) {
    span.setStatus(StatusCode.ERROR, e.getMessage());
    span.recordException(e);
} finally {
    span.end();
}

Span lifecycle:

Create span builder with a descriptive name
Add attributes before starting (optional)
Start the span
Execute your code
Record status and any exceptions
End the span (critical - resources not released until span ends)

Context and Scope

Context propagation connects related spans into a trace tree. When you make a span "current" using makeCurrent(), any child spans created inherit the trace context automatically.

java

import io.opentelemetry.context.Scope;

Span parentSpan = tracer.spanBuilder("parent_operation").startSpan();

try (Scope scope = parentSpan.makeCurrent()) {
    // Any spans created here become children of parentSpan
    processStep1();  // Creates child span automatically
    processStep2();  // Another child span

} finally {
    parentSpan.end();
}

The Scope returned by makeCurrent() must be closed to restore the previous context. Always use try-with-resources to ensure proper cleanup.

Context propagation across threads:

java

Context context = Context.current();

executor.submit(() -> {
    try (Scope scope = context.makeCurrent()) {
        // This code sees the same context as the parent thread
        childOperation();
    }
});

Creating Custom Spans

Manual spans allow you to instrument business logic and custom operations. This section demonstrates common span patterns for different scenarios.

Basic Span Pattern

The fundamental pattern for creating a span around a code block:

java

public void processUserRegistration(User user) {
    Span span = tracer.spanBuilder("user.registration")
        .setAttribute("user.id", user.getId())
        .setAttribute("user.country", user.getCountry())
        .startSpan();

    try (Scope scope = span.makeCurrent()) {
        // Business logic
        validateUser(user);
        createUserAccount(user);
        sendWelcomeEmail(user);

        span.setStatus(StatusCode.OK);

    } catch (Exception e) {
        span.setStatus(StatusCode.ERROR, e.getMessage());
        span.recordException(e);
        throw e;
    } finally {
        span.end();
    }
}

Key practices:

Use descriptive span names (e.g., user.registration not method1)
Add attributes before starting span when possible
Always set status (OK or ERROR)
Record exceptions for debugging
Use finally block to ensure span ends

Nested Spans

Create child spans for sub-operations to build a detailed execution tree:

java

public OrderSummary generateOrderSummary(String orderId) {
    Span parentSpan = tracer.spanBuilder("generate.order.summary")
        .setAttribute("order.id", orderId)
        .startSpan();

    try (Scope scope = parentSpan.makeCurrent()) {
        // Each method creates a child span
        Order order = fetchOrderDetails(orderId);
        parentSpan.setAttribute("order.total", order.getTotal());

        List<Item> items = fetchOrderItems(orderId);
        parentSpan.setAttribute("order.items.count", items.size());

        Customer customer = fetchCustomerInfo(order.getCustomerId());
        parentSpan.setAttribute("customer.tier", customer.getTier());

        parentSpan.setStatus(StatusCode.OK);
        return new OrderSummary(order, items, customer);

    } finally {
        parentSpan.end();
    }
}

private Order fetchOrderDetails(String orderId) {
    // Child span created automatically
    Span span = tracer.spanBuilder("fetch.order.details").startSpan();
    try (Scope scope = span.makeCurrent()) {
        // Database query here
        span.setStatus(StatusCode.OK);
        return orderRepository.findById(orderId);
    } finally {
        span.end();
    }
}

This creates a trace hierarchy:

text

generate.order.summary
├── fetch.order.details
├── fetch.order.items
└── fetch.customer.info

Span Attributes

Attributes add context to spans. Use them to record important business and technical information:

java

public void processPayment(PaymentRequest request) {
    Span span = tracer.spanBuilder("payment.process")
        // Business attributes
        .setAttribute("payment.method", request.getMethod())
        .setAttribute("payment.amount", request.getAmount())
        .setAttribute("payment.currency", request.getCurrency())
        .setAttribute("customer.id", request.getCustomerId())

        // Technical attributes
        .setAttribute("payment.gateway", "stripe")
        .setAttribute("payment.retry.attempt", 1)
        .startSpan();

    try (Scope scope = span.makeCurrent()) {
        PaymentResult result = paymentGateway.charge(request);

        // Add result attributes
        span.setAttribute("payment.transaction.id", result.getTransactionId());
        span.setAttribute("payment.status", result.getStatus());
        span.setAttribute("payment.processing.time.ms", result.getProcessingTimeMs());

        span.setStatus(StatusCode.OK);

    } finally {
        span.end();
    }
}

Attribute best practices:

Use semantic conventions when available (e.g., http.method, db.system)
Name attributes consistently (e.g., customer.id not customerId)
Avoid high-cardinality values (e.g., timestamps, UUIDs) unless necessary
Don't include sensitive data (PII, passwords, credit cards)

Span Events

Events mark important moments within a span's lifetime:

java

public ReportResult generateMonthlyReport(String reportId) {
    Span span = tracer.spanBuilder("generate.monthly.report")
        .setAttribute("report.id", reportId)
        .startSpan();

    try (Scope scope = span.makeCurrent()) {
        span.addEvent("report.data.fetch.started");
        List<Data> data = fetchReportData();
        span.addEvent("report.data.fetch.completed",
            Attributes.of(
                AttributeKey.longKey("data.rows"), data.size()
            ));

        span.addEvent("report.calculation.started");
        ReportResult result = calculateMetrics(data);
        span.addEvent("report.calculation.completed");

        span.addEvent("report.export.started");
        exportReport(result);
        span.addEvent("report.export.completed",
            Attributes.of(
                AttributeKey.stringKey("export.format"), "PDF",
                AttributeKey.longKey("export.size.kb"), result.getSizeKb()
            ));

        span.setStatus(StatusCode.OK);
        return result;

    } finally {
        span.end();
    }
}

Events help you understand the sequence and timing of operations within a single span. They're especially useful for debugging performance issues.

Custom Metrics

Metrics complement traces by aggregating data over time. Use metrics to track business KPIs, performance indicators, and system health.

Counter Pattern

Counters track cumulative totals that only increase. Use them for counting events like orders processed, API calls, or errors.

java

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.metrics.LongCounter;
import io.opentelemetry.api.metrics.Meter;
import io.opentelemetry.api.common.Attributes;
import io.opentelemetry.api.common.AttributeKey;

@Service
public class OrderMetricsService {

    private final LongCounter ordersCreated;
    private final LongCounter ordersCancelled;

    public OrderMetricsService() {
        Meter meter = GlobalOpenTelemetry.getMeter("com.example.orders");

        this.ordersCreated = meter
            .counterBuilder("orders.created")
            .setDescription("Total orders created")
            .setUnit("orders")
            .build();

        this.ordersCancelled = meter
            .counterBuilder("orders.cancelled")
            .setDescription("Total orders cancelled")
            .setUnit("orders")
            .build();
    }

    public void recordOrderCreated(String customerId, String productCategory) {
        ordersCreated.add(1, Attributes.of(
            AttributeKey.stringKey("customer.tier"), getCustomerTier(customerId),
            AttributeKey.stringKey("product.category"), productCategory
        ));
    }

    public void recordOrderCancelled(String cancellationReason) {
        ordersCancelled.add(1, Attributes.of(
            AttributeKey.stringKey("cancellation.reason"), cancellationReason
        ));
    }
}

Histogram Pattern

Histograms record distributions of values. Use them for measuring durations, sizes, or amounts.

java

@Service
public class PerformanceMetricsService {

    private final DoubleHistogram orderProcessingTime;
    private final DoubleHistogram orderValue;

    public PerformanceMetricsService() {
        Meter meter = GlobalOpenTelemetry.getMeter("com.example.performance");

        this.orderProcessingTime = meter
            .histogramBuilder("order.processing.duration")
            .setDescription("Order processing time")
            .setUnit("ms")
            .build();

        this.orderValue = meter
            .histogramBuilder("order.value")
            .setDescription("Order value distribution")
            .setUnit("USD")
            .build();
    }

    public void recordOrderProcessed(long durationMs, double orderValue, String country) {
        // Record processing time
        orderProcessingTime.record(durationMs, Attributes.of(
            AttributeKey.stringKey("country"), country
        ));

        // Record order value
        this.orderValue.record(orderValue, Attributes.of(
            AttributeKey.stringKey("country"), country
        ));
    }
}

Histograms allow you to calculate percentiles (p50, p95, p99) and analyze distribution patterns in your observability backend.

Observable Gauge Pattern

Gauges measure current values that can go up or down. Use observable gauges for measuring current state like active users, queue depth, or cache size.

java

@Component
public class SystemMetricsCollector {

    private final OrderQueueService queueService;
    private final CacheService cacheService;

    public SystemMetricsCollector(OrderQueueService queueService,
                                   CacheService cacheService) {
        this.queueService = queueService;
        this.cacheService = cacheService;

        Meter meter = GlobalOpenTelemetry.getMeter("com.example.system");

        // Observable gauge for current queue depth
        meter.gaugeBuilder("queue.depth")
            .setDescription("Current number of orders in queue")
            .setUnit("orders")
            .buildWithCallback(measurement -> {
                measurement.record(queueService.getCurrentDepth());
            });

        // Observable gauge for cache hit rate
        meter.gaugeBuilder("cache.hit.rate")
            .setDescription("Cache hit rate percentage")
            .setUnit("%")
            .buildWithCallback(measurement -> {
                double hitRate = cacheService.getHitRate() * 100;
                measurement.record(hitRate);
            });
    }
}

The callback function is invoked periodically (typically every 10-60 seconds) to collect the current value.

Integration with Spring Boot

Spring Boot applications can use manual instrumentation alongside automatic instrumentation. For complete Spring Boot setup with automatic instrumentation, see Spring Boot OpenTelemetry Integration.

This section shows different integration patterns for adding manual instrumentation to your Spring Boot applications.

Dependency Configuration

Add the OpenTelemetry API to your Spring Boot project:

xml

<!-- Maven pom.xml -->
<dependency>
    <groupId>io.opentelemetry</groupId>
    <artifactId>opentelemetry-api</artifactId>
    <version>1.56.0</version>
</dependency>

gradle

// Gradle build.gradle
dependencies {
    implementation 'io.opentelemetry:opentelemetry-api:1.56.0'
}

Using @WithSpan Annotation

The simplest way to add custom spans is using the @WithSpan annotation. This requires the Java Agent or Spring Boot Starter.

xml

<!-- Add annotation support -->
<dependency>
    <groupId>io.opentelemetry.instrumentation</groupId>
    <artifactId>opentelemetry-instrumentation-annotations</artifactId>
    <version>2.20.1-alpha</version>
</dependency>

java

import io.opentelemetry.instrumentation.annotations.WithSpan;
import io.opentelemetry.instrumentation.annotations.SpanAttribute;
import io.opentelemetry.api.trace.Span;

@Service
public class InventoryService {

    @WithSpan("inventory.check.availability")
    public boolean checkAvailability(
        @SpanAttribute("product.id") String productId,
        @SpanAttribute("quantity") int quantity
    ) {
        // Method automatically wrapped in span
        // Parameters become span attributes
        int available = getAvailableStock(productId);

        // Add custom attribute to current span
        Span.current().setAttribute("stock.available", available);

        return available >= quantity;
    }

    @WithSpan("inventory.reserve")
    public ReservationResult reserveItems(
        @SpanAttribute("product.id") String productId,
        @SpanAttribute("quantity") int quantity
    ) {
        // Span created automatically with method name
        return performReservation(productId, quantity);
    }
}

Benefits:

✅ Minimal code changes
✅ Automatic span lifecycle management
✅ Clean separation of business logic and instrumentation

Limitations:

❌ Less control over span details
❌ Cannot add events or complex attributes
❌ Requires Java Agent or Spring Boot Starter

Aspect-Oriented Programming

Use Spring AOP to add instrumentation without modifying business code:

java

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Span;
import io.opentelemetry.api.trace.Tracer;
import io.opentelemetry.api.trace.StatusCode;
import io.opentelemetry.context.Scope;
import org.aspectj.lang.ProceedingJoinPoint;
import org.aspectj.lang.annotation.Around;
import org.aspectj.lang.annotation.Aspect;
import org.springframework.stereotype.Component;

@Aspect
@Component
public class BusinessOperationTracing {

    private final Tracer tracer = GlobalOpenTelemetry.getTracer("business.aspects");

    @Around("@annotation(com.example.annotations.TracedOperation)")
    public Object traceBusinessOperation(ProceedingJoinPoint joinPoint) throws Throwable {
        String methodName = joinPoint.getSignature().getName();

        Span span = tracer.spanBuilder("business." + methodName)
            .setAttribute("class", joinPoint.getTarget().getClass().getSimpleName())
            .startSpan();

        try (Scope scope = span.makeCurrent()) {
            Object result = joinPoint.proceed();
            span.setStatus(StatusCode.OK);
            return result;

        } catch (Exception e) {
            span.setStatus(StatusCode.ERROR, e.getMessage());
            span.recordException(e);
            throw e;
        } finally {
            span.end();
        }
    }
}

// Custom annotation
@Target(ElementType.METHOD)
@Retention(RetentionPolicy.RUNTIME)
public @interface TracedOperation {
}

// Usage
@Service
public class PaymentService {

    @TracedOperation
    public PaymentResult processPayment(PaymentRequest request) {
        // Method automatically traced via AOP
        return chargeCustomer(request);
    }
}

Manual Tracer Injection

For maximum control, inject tracers as Spring beans:

java

import io.opentelemetry.api.GlobalOpenTelemetry;
import io.opentelemetry.api.trace.Tracer;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;

@Configuration
public class OpenTelemetryConfig {

    @Bean
    public Tracer applicationTracer() {
        return GlobalOpenTelemetry.getTracer(
            "com.example.myapp",
            "1.0.0"
        );
    }
}

// Use in services
@Service
public class OrderService {

    private final Tracer tracer;

    public OrderService(Tracer tracer) {
        this.tracer = tracer;
    }

    public OrderResult createOrder(OrderRequest request) {
        Span span = tracer.spanBuilder("order.create").startSpan();
        try (Scope scope = span.makeCurrent()) {
            // Your logic with full control
            return processOrder(request);
        } finally {
            span.end();
        }
    }
}

Key Takeaways

✓ Manual instrumentation captures business-specific operations that automatic instrumentation cannot detect.

✓ Use Tracer to create spans, add attributes for context, and record metrics for business KPIs.

✓ Context propagation with makeCurrent() automatically links child spans to parent spans.

✓ Spring Boot integration offers three approaches: @WithSpan annotations, AOP, or manual tracer injection.

✓ Follow best practices for naming, error handling, and resource management to avoid common pitfalls.