New Relic drinks straight from the Firehose: Consuming Amazon Kinesis data

February 9, 2024: Amazon Kinesis Data Firehose has been renamed to Amazon Data Firehose. Read the AWS What’s New post to learn more.

New Relic can now ingest data directly from Amazon Kinesis Data Firehose, expanding the insights New Relic can give you into your cloud stacks so you can deliver more perfect software. Kinesis Data Firehose is a fully managed service for delivering real-time streaming data to AWS services like Amazon Simple Storage Service (Amazon S3), Amazon Redshift, and a wide array of external destinations.

As software teams have been forced to adopt disconnected monitoring tools for their infrastructure, applications, logs, and digital experience, it has created data silos that result in blind spots. Blind spots increase the amount of work to switch between tools to uncover answers, and make it harder to diagnose issues. New Relic One provides a connected real-time view of all your data in one place. New Relic’s open platform is designed to ensure easy ingestion and analysis regardless of source.

With the release of Kinesis Data Firehose HTTP endpoint delivery, you can easily configure data streams to automatically ingest and forward data to New Relic. You can also configure Kinesis Data Firehose to transform your data before delivering it. You don’t need to write applications, manage resources, or create AWS Lambda functions, which makes it easier to manage and estimate costs for your data based on data volume.

In this post, we demonstrate how to stream Amazon CloudWatch Logs data to New Relic using a Firehose delivery stream. We show you how to create and configure a delivery stream to ingest and forward data to New Relic and configure CloudWatch Logs to write to Kinesis Data Firehose.

Prerequisites

Before continuing, you need a New Relic account and an Insights Insert API Key. You also need to install and configure the AWS Command Line Interface (AWS CLI) to make the policy and role changes covered by the post. For instructions, see Installing the AWS CLI.

You also need to make sure that your delivery stream has sufficient service-limit quotas to forward all your data. Kinesis Data Firehose has default quotas in place that range depending on the Region. You can create a case with AWS to request a quota increase.

Creating a delivery stream

To begin, you need to create a delivery stream to ingest logs from CloudWatch Logs. Complete the following steps:

Sign in to the AWS Management Console and navigate to Kinesis.
Under Data Firehose, choose Create delivery stream.
Enter a name for the delivery stream.
For Source, select Direct PUT or other sources.
Choose Next until you’re prompted to Select a destination and choose 3rd party partner.
From the drop-down menu, choose New Relic.
For New Relic HTTP API, add the following endpoint: https://aws-api.newrelic.com/firehose/v1.
Enter your Insights Insert API Key in the API access token field.
Configure common attributes.
Common attributes are inserted into every log that passes through the delivery stream, and can be queried against in New Relic Logs. As a best practice, we recommend including a logtype attribute to make sure your logs are properly parsed in New Relic Logs.The logtype attribute defined here appears in all your logs after they reach New Relic. If you intend to use the logtype attribute to determine which parsing rules are applied to your logs, we recommend creating a separate delivery stream for each logtype.
Configure and review the remaining settings as desired.

Validating the delivery stream configuration

Optionally, we recommend that you confirm that your delivery steam forwards logs to your New Relic account by completing the following steps:

On the Kinesis dashboard, choose Delivery Streams.
Choose the delivery stream you created in the previous section.
Expand Test with Demo Data and choose Start sending data.
Wait 3–5 minutes for demo data to be written to your delivery stream.
While you’re waiting, copy the Delivery Stream ARN—you need this to configure CloudWatch Logs to write to your delivery stream.If everything has been set up correctly, you should see demo data in your New Relic Logs account.
Choose Stop sending demo data to avoid incurring additional usage charges.

Configuring CloudWatch Logs to write to Kinesis Data Firehose

Your next step is to configure CloudWatch to write logs to Kinesis Data Firehose. For more information, see Subscription Filters with Amazon Kinesis Data Firehose. For this post, we configure our delivery stream to forward logs to New Relic instead of Amazon S3.

We begin by creating an AWS Identity and Access Management (IAM) role that allows CloudWatch Logs to write data to your delivery stream. You can do this using the AWS Command Line Interface (AWS CLI).

Use a text editor to create the following trust policy in a file (for example, ~/TrustPolicyForCWL.json). Make sure to replace us-east-1 with the Region containing your CloudWatch logs:
```
{
  "Statement": {
    "Effect": "Allow",
    "Principal": { "Service": "logs.us-east-1.amazonaws.com" },
    "Action": "sts:AssumeRole"
  }
}
```
Use the create-role command to create an IAM role using your newly created policy:

aws iam create-role \
      --role-name CWLtoKinesisFirehoseRole \
      --assume-role-policy-document file://~/TrustPolicyForCWL.json

Running the command returns the following code:

{
    "Role": {
        "AssumeRolePolicyDocument": {
            "Statement": {
                "Action": "sts:AssumeRole",
                "Effect": "Allow",
                "Principal": {
                    "Service": "logs.us-east-1.amazonaws.com"
                }
            }
        },
        "RoleId": "AAOIIAH450GAB4HC5F431",
        "CreateDate": "2020-07-14T13:46:29.431Z",
        "RoleName": "CWLtoKinesisFirehoseRole",
        "Path": "/",
        "Arn": "arn:aws:iam::123456789012:role/CWLtoKinesisFirehoseRole"
    }
}

Create a policy that allows CloudWatch to write logs to your delivery stream. As before, use a text editor to create a file (for example, ~/PermissionsForCWL.json) containing the following code:

{
    "Statement":[
      {
        "Effect":"Allow",
        "Action":["firehose:*"],
        "Resource":["arn:aws:firehose:region:123456789012:*"]
      },
      {
        "Effect":"Allow",
        "Action":["iam:PassRole"],
        "Resource":["arn:aws:iam::123456789012:role/CWLtoKinesisFirehoseRole"]
      }
    ]
}

Make sure to update the Region and AWS account ID placeholders in the preceding code with your account-specific details, and associate the policy with the role created at the beginning of this section. See the following code:

aws iam put-role-policy --role-name CWLtoKinesisFirehoseRole --policy-name Permissions-Policy-For-CWL --policy-document file://~/PermissionsForCWL.json

All required permissions should now be in place.

The last step is to create a CloudWatch Logs subscription filter that determines which logs are written to your delivery stream and forwarded to New Relic. See the following code:

aws logs put-subscription-filter \
    --log-group-name "syslog" \
    --filter-name "Destination" \
    --filter-pattern "ERROR" \
    --destination-arn "arn:aws:firehose:region:123456789012:deliverystream/my-delivery-stream" \
    --role-arn "arn:aws:iam::123456789012:role/CWLtoKinesisFirehoseRole"

In the preceding code, replace the value of destination-arn with the ARN of the delivery stream you created at the beginning of this post. You also need to update the role-arn with the ARN of the CWLtoKinesisFirehoseRole you created earlier.

After you complete these steps, you can confirm that data is flowing in New Relic Logs. The following screenshot shows Kinesis Data Firehose delivering data to New Relic Logs.

Conclusion

In this post, we showed you how to automatically ingest and forward CloudWatch Logs data into New Relic using an Kinesis Data Firehose HTTP endpoint. We hope you will use this knowledge to expand the use of data streams in your organization to deliver more perfect software faster.

The content and opinions in this post are those of the third-party author and AWS is not responsible for the content or accuracy of this post.

About the Authors

Mike Neville-O’Neill is a Product Manager on the New Relic Logging team. His professional interests include the ever-changing observability landscape, DevOps, information security, and log-based analytics

Colin Bookman is an ISV Sr. Solutions Architect at AWS based in Silicon Valley. He works with AWS ISVs and customers to help them build secure, high-performing, resilient, and cost-efficient infrastructure. He brings years of experience including childhood software and hardware projects, a Bachelor’s of Science in Computer Engineering from Georgia Tech, and past experience architecting and building services that handle millions of QPS. When not working, Colin enjoys taking his two dogs on long walks on the beach.

AWS Big Data Blog