OpenTelemetry Host Metrics receiver

hostmetricsreceiver is an OpenTelemetry Collector pluginopen in new window that gathers various metrics about the host system, for example, CPU, RAM, disk metrics and other system-level metrics.

By collecting and analyzing host metrics, you can gain insight into the performance and health of your host systems and identify potential problems or bottlenecks that could affect the overall performance of your applications and services.

What is OpenTelemetry Collector?

OpenTelemetry Collector is an agent that pulls telemetry data from systems you want to monitor and sends it to tracing toolsopen in new window using the OpenTelemetry protocol (OTLP).

OpenTelemetry Collector provides powerful data processing capabilities. It can aggregate, filter, transform, and enrich telemetry data as it flows through the system.

Host Metrics

To start collecting host metrics, you need to install Collector on each system you want to monitor and add the following lines to the Collector config:

processors:
  resourcedetection:
    detectors: [env, system]
  cumulativetodelta:

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      # CPU utilization metrics
      cpu:
      # Disk I/O metrics
      disk:
      # File System utilization metrics
      filesystem:
      # CPU load metrics
      load:
      # Memory utilization metrics
      memory:
      # Network interface I/O metrics & TCP connection metrics
      network:
      # Paging/Swap space utilization and I/O metrics
      paging:

service:
  pipelines:
    metrics:
      receivers: [otlp, hostmetrics]
      processors: [cumulativetodelta, batch, resourcedetection]
      exporters: [otlp/uptrace]

Filesystem metrics

If you are using unusual filesystems, you may want to configure filesystem receiver more thoroughly, for example, to scrape only supported filesystem types and avoid warnings:

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      cpu:
      disk:
      load:
      filesystem:
        include_fs_types:
          match_type: strict
          fs_types: [ext3, ext4]
      memory:
      network:
      paging:

Process metrics

To collect per process CPU, Memory, and Disk I/O metrics, you need to enable the respective scrapers:

receivers:
  hostmetrics:
    collection_interval: 10s
    scrapers:
      # Process count metrics
      process:
      # Per process CPU, Memory, and Disk I/O metrics
      processes:

Those scrapers are disabled by default, because they require running OpenTelemetry Collector with elevated permissions in order to access information about other processes.

On Linux, you can achieve that by running otelcol-contrib under root user:

# /lib/systemd/system/otelcol-contrib.service

User=root
Group=root

Or using sudo to start the process:

# /lib/systemd/system/otelcol-contrib.service

ExecStart=sudo /usr/bin/otelcol-contrib $OTELCOL_OPTIONS

Resource detection

The Resource Detection Processoropen in new window is responsible for detecting and setting resource information for telemetry data.

A resource represents a set of attributes that describe the entity producing the telemetry data. This can include information such as application name, version, environment, host name, and more

The resource detection processor can automatically discover resource attributes based on the environment in which it is running. For example, it can detect attributes from cloud providers, container orchestrators, or other runtime environments.

Here is how you can configure the processor in the OpenTelemetry Collector configuration file (otel-collector-config.yaml):

processors:
  resourcedetection:
    detectors: [env, system, ec2, gcp, azure]
    timeout: 5s
    # Override existing resource attributes if they already exist on incoming telemetry data.
    override: true

service:
  pipelines:
    traces:
      processors: [resourcedetection, batch]
    metrics:
      processors: [resourcedetection, batch]

By default, host.name is set to FQDN if possible, and a hostname provided by the OS is used as a fallback. This logic can be changed with the hostname_sources configuration, which is set to [dns, os] by default.

processors:
  resourcedetection/system:
    detectors: [system]
    system:
      hostname_sources: [os]

Container host metrics

On Linux, OpenTelemetry collects metrics from the Linux system directories. To collect metrics about the host system and not the container, you can mount the host filesystem when running the container:

# mount the entire filesystem
docker run -v /:/hostfs ...

# or mount only parts you need
docker run -v /proc:/hostfs/proc ...

Then configure root_path so the hostmetrics receiver knows where the root filesystem is:

receivers:
  hostmetrics:
    root_path: /hostfs

Exporting data to Uptrace

Uptrace is a OpenTelemetry APMopen in new window that supports distributed tracing, metrics, and logs. You can use it to monitor applications and troubleshoot issues.

Uptrace Overview

Uptrace comes with an intuitive query builder, rich dashboards, alerting rules with notifications, and integrations for most languages and frameworks.

Uptrace can process billions of spans and metrics on a single server and allows you to monitor your applications at 10x lower cost.

In just a few minutes, you can try Uptrace by visiting the cloud demoopen in new window (no login required) or running it locally with Dockeropen in new window. The source code is available on GitHubopen in new window.

To export host metrics data to Uptrace, follow the guideopen in new window to configure otlp/uptrace exporter.

What's next?

OpenTelemetry Collector supports a variety of receivers that allow you to collect telemetry data from different sources. Receivers act as plugins within the Collector and handle the ingestion of data from various protocols and formats. Here are some of the commonly used receivers available in the OpenTelemetry Collector:

Last Updated: 5/14/2024, 6:46:47 AM
Get insights and updates in your inbox: