Ingesting AWS CloudWatch metrics and logs

AWS CloudWatch allows to forward metrics and logs to third-party destinations using AWS Kinesis Data Firehose. Uptrace provides compatible HTTP endpoints for AWS Kinesis so you can monitor your AWS infrastructure with Uptrace.

• Metrics
  • yet-another-cloudwatch-exporter
  • Kinesis
• Logs
  • Terraform module
  • AWS Console
• Conclusion

Metrics

CloudWatch Metrics is a monitoring service provided by Amazon Web Services (AWS) that allows you to collect and track metrics from your applications and infrastructure resources in real-time.

You can configure AWS CloudWatch to send metrics to Uptrace using AWS Kinesis Data Firehose. It works well, but AWS CloudWatch does not support exporting tags and only provides access to standard dimensions such as InstanceId and InstanceType, but not InstanceName.

As a workaround, you can use yet-another-cloudwatch-exporter that provides CloudWatch metrics as Prometheus metrics with AWS tags as labels. This requires using Prometheus instead of AWS Kinesis.

yet-another-cloudwatch-exporter

1. First, install YACE by downloading a binary file or using Docker/Kubernetes.

   The exporter must run in an environment that has access to AWS. You can use the following IAM policy to grant all the permissions required by YACE:

   {
     "Version": "2012-10-17",
     "Statement": [
       {
         "Action": ["tag:GetResources", "cloudwatch:GetMetricData", "cloudwatch:GetMetricStatistics", "cloudwatch:ListMetrics", "apigateway:GET", "aps:ListWorkspaces", "autoscaling:DescribeAutoScalingGroups", "dms:DescribeReplicationInstances", "dms:DescribeReplicationTasks", "ec2:DescribeTransitGatewayAttachments", "ec2:DescribeSpotFleetRequests", "shield:ListProtections", "storagegateway:ListGateways", "storagegateway:ListTagsForResource", "iam:ListAccountAliases"],
         "Effect": "Allow",
         "Resource": "*"
       }
     ]
   }
   

2. Next, configure YACE using a YAML configuration file. To specify which configuration file to load, pass the -config.file flag on the command line.

   YACE supports automatic resource discovery via tags, but you can also use static and custom namespace jobs.

   Here is an example config file for EC2, but you can find more on GitHub:

   apiVersion: v1alpha1
   discovery:
     jobs:
       - type: AWS/EC2
         regions:
           - us-east-1
         period: 300
         length: 300
         metrics:
           - name: CPUUtilization
             statistics: [Average]
           - name: NetworkIn
             statistics: [Average, Sum]
           - name: NetworkOut
             statistics: [Average, Sum]
           - name: NetworkPacketsIn
             statistics: [Sum]
           - name: NetworkPacketsOut
             statistics: [Sum]
           - name: DiskReadBytes
             statistics: [Sum]
           - name: DiskWriteBytes
             statistics: [Sum]
           - name: DiskReadOps
             statistics: [Sum]
           - name: DiskWriteOps
             statistics: [Sum]
           - name: StatusCheckFailed
             statistics: [Sum]
           - name: StatusCheckFailed_Instance
             statistics: [Sum]
           - name: StatusCheckFailed_System
             statistics: [Sum]
   

3. Once you have YACE running, the Prometheus metrics should be available at http://localhost:5000/metrics.

   Now you need to add a corresponding job to your Prometheus configuration:

   - job_name: 'yet-another-cloudwatch-exporter'
     metrics_path: '/metrics'
     static_configs:
       - targets: ['localhost:5000']
   

4. The final step is to configure Prometheus to export data to Uptrace using remote write or OpenTelemetry Collector. You can also use Grafana integration to explore collected Prometheus metrics and create dashboards provided by YACE.

Kinesis

If you don't want to configure Prometheus with yet-another-cloudwatch-exporter, you can configure AWS Kinesis using a terraform module or AWS console.

Terraform module

Uptrace provides a Terraform module that configures AWS CloudWatch to send metrics to Uptrace. Refer to the module's readme for details.

AWS Console

You can also configure CloudWatch manually using the AWS Console.

1. Create a new Kinesis Data Firehose Delivery Stream with the following details:

   • Stream source: Direct PUT
   • Kinesis endpoint: https://api.uptrace.dev/api/v1/cloudwatch/metrics
   • API Key: Enter the Uptrace DSN for your project.
   • Content Encoding: GZIP.

2. Create a new CloudWatch Metric Stream.

   1. Open the CloudWatch AWS console.
   2. Choose Metrics → Streams.
   3. Click the Create metric stream button.
   4. Choose CloudWatch metric namespaces to include in the metric stream.
   5. Choose Select an existing Firehose owned by your account, and select the Firehose Delivery Stream you created earlier.
   6. Specify an Output Format of json.
   7. Optionally, specify a name for this metric stream under Metric Stream Name.
   8. Click on the Create metric stream button.

Logs

CloudWatch Logs is a log management service provided by Amazon Web Services (AWS) that allows you to collect, monitor, and analyze log files from your applications and infrastructure.

You can configure AWS CloudWatch to send logs to Uptrace using AWS Kinesis Data Firehose.

Terraform module

Uptrace provides a Terraform module that configures AWS CloudWatch to send logs to Uptrace. Refer to the module's readme for details.

AWS Console

You can also configure CloudWatch manually using the AWS Console.

1. Create a new Kinesis Data Firehose Delivery Stream with the following details:

   • Stream source: Direct PUT
   • Kinesis endpoint: https://api.uptrace.dev/api/v1/cloudwatch/logs
   • API Key: Enter the Uptrace DSN for your project.
   • Content Encoding: GZIP.

2. Create a new CloudWatch log group using the Firehose Delivery Stream you created earlier.

Conclusion

AWS CloudWatch is a powerful tool for monitoring, collecting, and analyzing data from your AWS resources and applications. It helps you identify and resolve issues, optimize resource utilization, and ensure the overall performance and availability of your AWS infrastructure.

Using AWS Kinesis Data Firehose, you can forward metrics and logs to Uptrace, open source APM tool that supports distributed tracing, metrics, and logs.

• OpenTelemetry Go Lambda
• OpenTelemetry Node.js Lambda