OpenTelemetry Collector
OpenTelemetry Collector is a high-performance, scalable, and reliable data collection pipeline for observability data. It receives telemetry data from various sources, performs processing and translation to a common format, and then exports the data to various backends for storage and analysis.
Otel Collector supports multiple data formats, protocols, and platforms, making it a flexible and scalable solution for observability needs.
How OpenTelemetry Collector works?
OpenTelemetry Collector is a vendor-agnostic proxy/middleman between your application and a distributed tracing tool such as Uptrace or Jaeger.
OpenTelemetry Collector works by receiving telemetry data from various sources, processing and normalizing the data, and then exporting it to various backends for storage and analysis.
OpenTelemetry Collector provides a flexible and extensible platform for telemetry data management that can be tailored to meet the specific needs of different environments and use cases.
Otel Collector is written in Go and licensed under Apache 2.0 license which allows you to change the source code and install custom extensions. That comes at a cost of running and maintaining your own OpenTelemetry Collector instances.
When to use OpenTelemetry Collector?
Most of the time, sending telemetry data directly to a backend is the great way to get started with OpenTelemetry. But you may want to deploy a collector alongside your services to get batching, retries, sensitive data filtering, and more.
The most prominent OpenTelemetry Collector feature is the ability to operate on whole traces instead of individual spans. To achieve that, OpenTelemetry Collector buffers the received spans and groups them by a trace id. That is the key requirement to implement tail-based sampling.
OpenTelemetry Collector can also act as an agent that pulls telemetry data from monitored systems, for example, Redis or filesystem metrics.
otelcol vs otelcol-contrib
OpenTelemetry Collector has 2 repositories on GitHub:
opentelemetry-collector is the core that contains only the most crucial components. It is distributed as
otelcolbinary.opentelemetry-collector-contrib contains the core and all additional available components, for example, Redis and PostgreSQL receivers. It is distributed as
otelcol-contribbinary.
You should always install and use the otelcol-contrib, because it is as stable as the core and supports more features.
Installation
OpenTelemetry Collector distributes pre-compiled binaries for Linux, MacOS, and Windows.
Linux
To install otelcol-contrib binary with the associated systemd service, run the following command replacing 0.56.0 with the desired version and amd64 with the desired architecture:
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.56.0/otelcol-contrib_0.56.0_linux_amd64.deb
sudo dpkg -i otelcol-contrib_0.56.0_linux_amd64.deb
wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.56.0/otelcol_0.56.0_linux_amd64.rpm
sudo rpm -ivh otelcol_0.56.0_linux_amd64.rpm
You can check the status of the installed service with:
sudo systemctl status otelcol-contrib
And check the logs with:
sudo journalctl -u otelcol-contrib -f
You can edit the config at /etc/otelcol-contrib/config.yaml and restart OpenTelemetry Collector:
sudo systemctl restart otelcol-contrib
Compiling from sources
You can also compile OpenTelemetry Collector locally:
git clone https://github.com/open-telemetry/opentelemetry-collector-contrib.git
cd opentelemetry-collector-contrib
make install-tools
make otelcontribcol
./bin/otelcontribcol_linux_amd64 --config ./examples/local/otel-config.yaml
Configuration
By default, you can find the config file at /etc/otelcol-contrib/config.yaml, for example:
TIP
Don't repeat a common mistake by configuring a receiver or an exporter without adding it to service.pipelinessection. Such receivers and exporters are silently ignored.
# receivers configure how data gets into the Collector.
receivers:
otlp:
protocols:
grpc:
http:
# processors specify what happens with the received data.
processors:
resourcedetection:
detectors: [system]
batch:
send_batch_size: 10000
timeout: 10s
# exporters configure how to send processed data to one or more backends.
exporters:
otlp/uptrace:
endpoint: otlp.uptrace.dev:4317
headers:
uptrace-dsn: 'https://<token>@uptrace.dev/<project_id>'
# service.pipelines pull the configured receivers, processors, and exporters together into
# pipelines that process data.
#
# receivers, processors, and exporters that are not used in pipelines are silently ignored.
service:
pipelines:
traces:
receivers: [otlp]
processors: [batch]
exporters: [otlp/uptrace]
metrics:
receivers: [otlp]
processors: [batch, resourcedetection]
exporters: [otlp/uptrace]
logs:
receivers: [otlp]
processors: [batch]
exporters: [otlp/uptrace]
You can use Collector Configurator to generate the config file or learn more about Collector using the official documentation.
Extensions
Extensions provide additional capabilities for OpenTelemetry Collector and do not require direct access to telemetry data, for example, Health Check extension responds to health check requests.
extensions:
# Health Check extension responds to health check requests
health_check:
# PProf extension allows fetching Collector's performance profile
pprof:
# zPages extension enables in-process diagnostics
zpages:
# Memory Ballast extension configures memory ballast for the process
memory_ballast:
size_mib: 512
Prometheus integration
See OpenTelemetry Collector Prometheus.
Exporting data to Uptrace
You can configure OpenTelemetry Collector to export data to Uptrace using the OTLP exporter:
exporters:
otlp:
endpoint: otlp.uptrace.dev:4317
headers:
# Copy your project DSN here
uptrace-dsn: 'https://<token>@uptrace.dev/<project_id>'
Host Metrics
hostmetricsreceiver is a Collector plugin that gathers various metrics about the host system, for example, CPU, RAM, disk metrics and more.
To start collecting host metrics, you need to install Collector on each system you want to monitor and add the following lines to the Collector config:
processors:
resourcedetection:
detectors: [env, system]
receivers:
hostmetrics:
collection_interval: 10s
scrapers:
cpu:
disk:
filesystem:
load:
memory:
network:
paging:
service:
pipelines:
metrics:
receivers: [otlp, hostmetrics]
processors: [batch, resourcedetection]
exporters: [otlp/uptrace]
If you are using unusual filesystems, you may want to configure the receiver more thoroughly, for example, to scrape only supported filesystems:
receivers:
hostmetrics:
collection_interval: 10s
scrapers:
cpu:
disk:
load:
filesystem:
include_fs_types:
match_type: strict
fs_types: [ext3, ext4]
memory:
network:
paging:
Resource Detection
To detect resource information from the host, Otel Collector comes with resourcedetection.
Resource Detection Processor automatically detects and labels metadata about the environment in which the data was generated. Such metadata, known as "resources", provides context to the telemetry data and can include information such as the host, service, container, and cloud provider.
For example, to detect host.name and os.type attributes, you can use system detector:
processors:
resourcedetection:
detectors: [env, system]
service:
pipelines:
metrics:
receivers: [otlp, hostmetrics]
processors: [batch, resourcedetection]
exporters: [otlp/uptrace]
To detect more information, you can use more specialized detectors, for example, if you are using Amazon EC2, you can use ec2 detector to also discover cloud.region and cloud.availability_zone attributes:
processors:
resourcedetection/ec2:
detectors: [env, ec2]
If you are using Google Cloud:
processors:
resourcedetection/gcp:
detectors: [env, gcp]
If you are using Docker:
processors:
resourcedetection/docker:
detectors: [env, docker]
You can check the official documentation to learn about available detectors for Heroku, Azure, Consul, and many others.
Memory Limiter
memorylimiterprocessor is a component that allows users to limit the amount of memory consumed by the OpenTelemetry Collector when processing telemetry data. It prevents the collector from using too much memory, which can lead to performance issues or even crashes.
Memory Limiter Processor works by periodically checking the amount of memory consumed by the OpenTelemetry Collector and comparing it to a user-defined memory limit. If the collector is using more memory than the specified limit, the processor will start dropping telemetry data until the memory usage falls below the limit.
To enable memory limiter:
processors:
memory_limiter:
check_interval: 1s
limit_mib: 4000
spike_limit_mib: 800
service:
pipelines:
metrics:
processors: [memory_limiter]
Uptrace
Uptrace is an open source DataDog competitor with an intuitive query builder, rich dashboards, alerting rules, and integrations for most languages and frameworks. It can process billions of spans and metrics on a single server and allows to monitor your applications at 10x lower cost.
Uptrace uses ClickHouse database to store traces, metrics, and logs. You can use it to monitor applications and set up automatic alerts to receive notifications via email, Slack, Telegram, and more.
You can get started with Uptrace by downloading a DEB/RPM package or a pre-compiled Go binary.
See also
