AWS CloudWatch metrics and logs

AWS CloudWatch allows to forward metrics and logs to third-party destinations using AWS Data Firehose. Uptrace provides compatible HTTP endpoints for AWS Data Firehost so you can monitor your AWS infrastructure with Uptrace.

flowchart LR cloudwatch([AWS CloudWatch]) firehose([AWS Data Firehose]) uptrace([Uptrace endpoints]) cloudwatch --> firehose firehose -->|HTTP| uptrace

Metrics

CloudWatch Metrics is a monitoring service provided by Amazon Web Services (AWS) that allows you to collect and track metrics from your applications and infrastructure resources in real-time.

You can configure AWS CloudWatch to send metrics to Uptrace using AWS Data Firehose. It works well, but AWS CloudWatch does not support exporting tags and only provides access to standard dimensions such as InstanceId and InstanceType, but not InstanceName.

As a workaround, you can use yet-another-cloudwatch-exporter that provides CloudWatch metrics as Prometheus metrics with AWS tags as labels. This requires using Prometheus instead of AWS Data Firehose.

yet-another-cloudwatch-exporter

First, install YACE by downloading a binary file or using Docker/Kubernetes.
The exporter must run in an environment that has access to AWS. You can use the following IAM policy to grant all the permissions required by YACE:

json

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Action": [
        "tag:GetResources",
        "cloudwatch:GetMetricData",
        "cloudwatch:GetMetricStatistics",
        "cloudwatch:ListMetrics",
        "apigateway:GET",
        "aps:ListWorkspaces",
        "autoscaling:DescribeAutoScalingGroups",
        "dms:DescribeReplicationInstances",
        "dms:DescribeReplicationTasks",
        "ec2:DescribeTransitGatewayAttachments",
        "ec2:DescribeSpotFleetRequests",
        "shield:ListProtections",
        "storagegateway:ListGateways",
        "storagegateway:ListTagsForResource",
        "iam:ListAccountAliases"
      ],
      "Effect": "Allow",
      "Resource": "*"
    }
  ]
}

Next, configure YACE using a YAML configuration file. To specify which configuration file to load, pass the -config.file flag on the command line.
YACE supports automatic resource discovery via tags, but you can also use static and custom namespace jobs.
Here is an example config file for EC2, but you can find more on GitHub:

yaml

apiVersion: v1alpha1
discovery:
  jobs:
    - type: AWS/EC2
      regions:
        - us-east-1
      period: 300
      length: 300
      metrics:
        - name: CPUUtilization
          statistics: [Average]
        - name: NetworkIn
          statistics: [Average, Sum]
        - name: NetworkOut
          statistics: [Average, Sum]
        - name: NetworkPacketsIn
          statistics: [Sum]
        - name: NetworkPacketsOut
          statistics: [Sum]
        - name: DiskReadBytes
          statistics: [Sum]
        - name: DiskWriteBytes
          statistics: [Sum]
        - name: DiskReadOps
          statistics: [Sum]
        - name: DiskWriteOps
          statistics: [Sum]
        - name: StatusCheckFailed
          statistics: [Sum]
        - name: StatusCheckFailed_Instance
          statistics: [Sum]
        - name: StatusCheckFailed_System
          statistics: [Sum]

Once you have YACE running, the Prometheus metrics should be available at http://localhost:5000/metrics.
Now you need to add a corresponding job to your Prometheus configuration:
yaml
```
- job_name: 'yet-another-cloudwatch-exporter'
  metrics_path: '/metrics'
  static_configs:
    - targets: ['localhost:5000']
```
The final step is to configure Prometheus to export data to Uptrace using remote write or OpenTelemetry Collector. You can also use Grafana integration to explore collected Prometheus metrics and create dashboards provided by YACE.

AWS Data Firehose

If you don't want to configure Prometheus with yet-another-cloudwatch-exporter, you can configure AWS Data Firehose using a terraform module or AWS console.

Terraform module

Uptrace provides a Terraform module that configures AWS CloudWatch to send metrics to Uptrace. Refer to the module's readme for details.

AWS Console

You can also configure CloudWatch manually using the AWS Console.

Create a new Data Firehose Delivery Stream with the following details:
- Stream source: Direct PUT
- Endpoint: https://api.uptrace.dev/api/v1/cloudwatch/metrics
- API Key: Enter the Uptrace DSN for your project.
- Content Encoding: GZIP.
Create a new CloudWatch Metric Stream.
1. Open the CloudWatch AWS console.
2. Choose Metrics → Streams.
3. Click the Create metric stream button.
4. Choose CloudWatch metric namespaces to include in the metric stream.
5. Choose Select an existing Firehose owned by your account, and select the Firehose Delivery Stream you created earlier.
6. Specify an Output Format of json.
7. Optionally, specify a name for this metric stream under Metric Stream Name.
8. Click on the Create metric stream button.

Logs

CloudWatch Logs is a log management service provided by Amazon Web Services (AWS) that allows you to collect, monitor, and analyze log files from your applications and infrastructure.

You can configure AWS CloudWatch to send logs to Uptrace using AWS Data Firehose.

Terraform module

Uptrace provides a Terraform module that configures AWS CloudWatch to send logs to Uptrace. Refer to the module's readme for details.

AWS Console

You can also configure CloudWatch manually using the AWS Console.

Create a new Data Firehose Delivery Stream with the following details:
- Stream source: Direct PUT
- Endpoint: https://api.uptrace.dev/api/v1/cloudwatch/logs
- API Key: Enter the Uptrace DSN for your project.
- Content Encoding: GZIP.
Create a new CloudWatch log group using the Firehose Delivery Stream you created earlier.

Conclusion

AWS CloudWatch is a powerful tool for monitoring, collecting, and analyzing data from your AWS resources and applications. It helps you identify and resolve issues, optimize resource utilization, and ensure the overall performance and availability of your AWS infrastructure.

Using AWS Data Firehose, you can forward metrics and logs to Uptrace, open source APM tool that supports distributed tracing, metrics, and logs.

OTel Collector

PrometheusThere are 2 ways to ingest Prometheus metrics into Uptrace: