AWS CloudWatch Logs

Overview

There are two approaches to send AWS logs to Dash0 via Firehose:

1. CloudWatch Logs Subscription Filters (Standard approach): CloudWatch Logs events can be sent to Firehose using CloudWatch subscription filters. This approach works with all CloudWatch Log groups and retains logs in AWS.

2. Lambda Direct Logging to Firehose: AWS Lambda supports sending logs directly to Firehose without storing them in CloudWatch Logs. This approach is more cost-effective but only works with Lambda functions.

This guide will show you how to set up both approaches.

Choosing the Right Approach

Feature	CloudWatch Logs Subscription Filters	Lambda Direct Logging to Firehose
Supported Services	All AWS services that use CloudWatch Logs	Lambda functions only
Cost	Standard CloudWatch Logs pricing	50% less on ingest, no storage costs
Log Retention in AWS	Yes, logs are stored in CloudWatch	No, logs are not stored after delivery
CloudWatch Features	Full access to CloudWatch Logs features (Logs Insights, Live Tail, etc.)	No access to CloudWatch Logs Standard features
Setup Complexity	More complex (requires subscription filters and IAM roles)	Simpler (direct configuration in Lambda)

When to use CloudWatch Logs Subscription Filters:

• You need to retain logs in AWS for compliance or auditing purposes
• You need CloudWatch Logs features like Logs Insights or Live Tail
• You're collecting logs from non-Lambda AWS services

When to use Lambda Direct Logging to Firehose:

• You want to reduce AWS logging costs (50% less on ingest, no storage costs)
• You don't need to retain logs in AWS
• You don't need CloudWatch Logs features like Logs Insights or Live Tail
• You're only collecting logs from Lambda functions

Create a Firehose delivery stream

Follow AWS documentation to create a Firehose delivery stream from AWS console:

1. Choose source and destination

Choose "Direct PUT" as the source and "HTTP Endpoint" as the destination of your Firehose stream.

2. Firehose stream name

Give your Firehose stream a name.

3. Destination settings

In HTTP Endpoint URL, enter AWS CloudWatch Logs via Firehose HTTP endpoint, which can be found in your Dash0 organization settings under Endpoints:

Create a Dash0 authentication token for the Firehose stream. By default, data will be sent to the default dataset. If you want to send the data to another dataset, you can restrict the token access to that specific dataset. It's also recommended to grant only the ingestion permission to this token.

In the authentication section, you can either choose "Use access key" option and enter the authentication token directly as the Access Key:

or use AWS Secrets Manager to retrieve the token programmatically.

In the Parameters section, you can add additional attributes to the log events. For example, you can add cloud.region as key and specify the region of your firehose stream such as eu-west-1 as value. This attribute will be added to the log group resource in Dash0 and allows to distinguish log groups from different regions.

You can keep the rest of the settings as default. Data sent from CloudWatch Logs to Amazon Data Firehose is already compressed with gzip level 6 compression, so you do not need to use compression within your Firehose delivery stream.

4. Backup settings

Create or choose an existing S3 bucket to store data in case of delivery failures.

Create a CloudWatch subscription filter to send logs to Firehose

Follow AWS documentation to create a CloudWatch subscription filter that sends any incoming log events that match your defined filters to your Firehose delivery stream previously created. You can skip step 1-7 in the AWS documentation which are about creating a Firehose delivery stream and go directly to step 8.

1. Create an IAM role to set up permissions for CloudWatch Logs to send data to Firehose

Step 8-11 in the AWS documentation will guide you through creating such IAM role through the command line.

If you are using the AWS Management Console, you can go to IAM > Roles and create a new role.

Choose Custom trust policy and update the trust policy statement with:

Note: Make sure to replace <REGION> and <ACCOUNT_ID> with your AWS region and account ID.

Click Next to go to the Add Permissions step. You can either select AmazonKinesisFirehoseFullAccess policy or create a custom policy which only grants firehose:PutRecord permission to the role with the following statement.

Note: Make sure to replace <REGION>, <ACCOUNT_ID> and <DELIVERY_STREAM_NAME> with your AWS region, account ID and Firehose delivery stream name.

At the final step, name the role and click Create role to complete the setup.

2. Create a subscription filter for the log group

In the AWS console, go to CloudWatch > Logs > Log groups and select the log group you want to send to Firehose. Click Actions > Create subscription filter > Create Amazon Data Firehose Subscription filter.

In Choose destination, select the Firehose delivery stream you created in the previous step.

In Grant permission, select the IAM role you created in the previous step.

Choose the log format and filter pattern for the log data you want to send to Firehose.

Enable System Fields for Multi-Account/Region Attribution (Recommended)

When sendings logs into a Firehose delivery stream, you should enable system fields to ensure proper resource attribution in Dash0. This adds cloud.account.id and cloud.region attributes to your spans, making it easy to identify the source account and region for each log and helps with detecing the right resource who emitted the log.

Set up Lambda Direct Logging to Firehose

AWS Lambda supports sending logs directly to Firehose without storing them in CloudWatch Logs. This approach is more cost-effective (50% less on ingest, no storage costs) but only works with Lambda functions and doesn't support CloudWatch Logs Standard features like Logs Insights and Live Tail.

Prerequisites:

• A Firehose delivery stream configured to send data to Dash0 (follow the "Create a Firehose delivery stream" section above)
• Lambda functions that you want to send logs from

Steps to configure Lambda direct logging to Firehose:

1. Navigate to the Lambda console

Go to the AWS Lambda console and select or create a function to set up Firehose as the logging destination.

2. Configure logging settings

1. In the Lambda function page, select the Configuration tab
2. Select Monitoring and operations tools on the left pane
3. In the Logging configuration section, click Edit

3. Select Firehose as the log destination

1. In the Log destination section, select Amazon Data Firehose (Amazon CloudWatch Logs is the default selection)
2. Under CloudWatch delivery log group, choose either Create new log group or Existing log group

4. Configure the delivery log group

If creating a new delivery log group:

1. Enter a log group name
2. Select the Firehose delivery stream you created earlier
3. Provide an IAM role for CloudWatch Logs to deliver logs to Firehose (you can use the same role created in the previous section)

If using an existing delivery log group:

1. Select a delivery log group from the dropdown
2. The selected delivery log group must have a configured Firehose destination and match the destination you selected

5. Configure advanced logging options (optional)

You can configure advanced logging controls for your Lambda function:

• Select JSON structured format for easier parsing in Dash0
• Set log level filters for both application and system logs
• Configure other logging options as needed

6. Save your configuration

Click Save to apply your changes. Your Lambda function will now send logs directly to Firehose, which will deliver them to Dash0.

Explore logs in Dash0

Once everything is set up, you can find CloudWatch logs by using the built-in AWS CloudWatch logging view.

Additional Considerations

Cost Comparison

The Lambda direct logging to Firehose approach offers significant cost savings:

• 50% lower ingest costs compared to standard CloudWatch Logs
• No storage costs as logs are not retained in AWS after delivery to Dash0
• Simplified architecture with direct delivery from Lambda to Firehose

Feature Limitations

When using Lambda direct logging to Firehose, be aware of these limitations:

• No access to CloudWatch Logs Standard features (Logs Insights, Live Tail)
• Only works with Lambda functions, not other AWS services
• Logs are not retained in AWS for historical analysis or compliance purposes

Best Practices

• Use structured logging (JSON format) for better searchability in Dash0
• Configure appropriate log levels to control the volume of logs sent to Dash0
• Set up error handling and monitoring for the Firehose delivery stream
• Consider using both approaches in combination if you need to retain some logs in AWS while optimizing costs for others