How do I add Amazon EC2 metadata when pushing logs to Kinesis using Kinesis Agent?

5 minute read
0

I'm trying to send logs from Amazon Elastic Compute Cloud (Amazon EC2) to Amazon Kinesis using Amazon Kinesis Agent. How do I append the Amazon EC2 metadata to each log line?

Short description

To append the EC2 metadata to each log line, do the following:

1.    Install and set up Amazon Kinesis Agent on either a Linux or Windows platform.

2.    Update your configuration settings to display EC2 metadata.

3.    Verify that you have the required AWS Identity and Access Management (IAM) permissions.

Note: You need IAM permissions to retrieve EC2 metadata and to publish data to Amazon Kinesis Data Firehose.

Resolution

On a Linux platform

To use Amazon Kinesis Agent to send logs to Kinesis Data Firehose, do the following:

1.    Download and install the agent.

2.    Create an IAM role with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:PutRecord",
        "firehose:PutRecordBatch"
      ],
      "Resource": ["<FIREHOSE_ARN>"
      ]
    },
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceAttribute",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeTags"
      ],
      "Resource": "<EC2_ARN>"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "cloudwatch:PutMetricData",
      "Resource": "*"
    }
  ]
}

Note: For Firehose_ARN and EC2_ARN, enter your Amazon Resource Names (ARN) information. Or, you can use a wildcard "*". For more information, see Amazon Resource Names (ARNs) for Amazon EC2.

3.    Attach the newly created IAM role to the EC2 instances where you installed Amazon Kinesis Agent. For more information about assigning an existing IAM role, see How do I assign an existing IAM role to an EC2 instance?

4.    Edit the /etc/aws-kinesis/agent.json file:

{
  "cloudwatch.emitMetrics": true,
  "kinesis.endpoint": "",
  "firehose.endpoint": "firehose.us-east-1.amazonaws.com",
  
  "flows": [
    {
      "filePattern": "/tmp/app.log*",
      "deliveryStream": "yourdeliverystream",
      "dataProcessingOptions": [
          {
                    "optionName": "LOGTOJSON",
                    "logFormat": "COMMONAPACHELOG"
              },
          {
            "optionName": "ADDEC2METADATA",
        "logFormat": "COMMONAPACHELOG"
          }
      ]
    }
  ]
}

In this example, the /etc/aws-kinesis/agent.json file processes the COMMONAPACHELOG log file format. Note that if your log file is in a different format, then the dataProcessingOptions setting must be updated to your log file format. For more information about agents and processing options, see Use the agent to Preprocess Data.

Important: Add ADDEC2METADATA for the optionName field to be sure that the EC2 metadata is appended to each log line. By default, the Amazon Kinesis Agent appends the following parameters in EC2 metadata: privateIp, availabilityZone, instanceId, instanceType, accountId, amiId, region, and metadataTimestamp.

5.    Configure and start the agent. The agent now runs as a system service in the background. It continuously monitors the specified files, sending data to the specified delivery stream. The agent activity is logged in the /var/log/aws-kinesis-agent/aws-kinesis-agent.log file, similar to this example output:

{
    "host": "157.92.12.106",
    "ident": null,
    "authuser": null,
    "datetime": "31/Aug/1995:20:50:31 -0400",
    "request": "GET /history/astp/astp-spacecraft.txt HTTP/1.0",
    "response": "200",
    "bytes": "440",
    "privateIp": "X.X.X.X",
    "availabilityZone": "us-east-1c",
    "instanceId": "i-01bxxxxxxxxxx43a0",
    "instanceType": "t2.xlarge",
    "accountId": "585xxxxxx740",
    "amiId": "ami-0fc61db8544a617ed",
    "region": "us-east-1",
    "metadataTimestamp": "2020-04-20T02:28:40+0000"
  }

On a Windows platform

To use Amazon Kinesis Tap Agent to send data to Kinesis Data Firehose, perform the following steps:

1.    Install Amazon Kinesis Agent for Microsoft Windows.

2.    Create an IAM role with the following permissions:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "firehose:DeleteDeliveryStream",
        "firehose:PutRecord",
        "firehose:PutRecordBatch",
        "firehose:UpdateDestination"
      ],
      "Resource": ["<KFH_ARN>"
      ]
    },
    {
      "Sid": "VisualEditor0",
      "Effect": "Allow",
      "Action": [
        "ec2:DescribeInstances",
        "ec2:DescribeInstanceAttribute",
        "ec2:DescribeInstanceTypes",
        "ec2:DescribeInstanceStatus",
        "ec2:DescribeTags"
      ],
      "Resource": "<EC2_ARN>"
    },
    {
      "Sid": "VisualEditor1",
      "Effect": "Allow",
      "Action": "cloudwatch:PutMetricData",
      "Resource": "*"
    }
  ]
}

Note: For KFH_ARN, and EC2_ARN, enter your Amazon Resource Names (ARN) information. For more information, see Amazon Resource Names (ARNs) for Amazon EC2.

3.    Attach the newly created IAM role to the EC2 instances where you installed Kinesis Tap Agent.

4.    Open the C:\Program Files\Amazon\AWSKinesisTap\appsettings.json file:

{
  "Sources": [
    {
      "Id": "W3SVCLog1",
      "SourceType": "W3SVCLogSource",
      "Directory": "C:\\inetpub\\logs\\LogFiles\\W3SVC1",
      "FileNameFilter": "*.log",
      "TimeZoneKind": "UTC"
    }
  ],
  "Sinks": [
    {
      "Id": "W3SVCLogSink",
      "SinkType": "KinesisFirehose",
      "Region": "eu-central-1",
      "StreamName": " W3SVCLogStream",
  "Format": "json",
  <b>"ObjectDecoration": "instance_id={instance_id};hostname={hostname};ec2:local-hostname={ec2:local-hostname};computername={computername};env:computername={env:computername};timestamp:yyyyMMdd={timestamp:yyyyMMdd}"</b>
    }    
  
  ],
  "Pipes": [
    {
      "Id": "W3SVCLog1ToKinesisStream",
      "SourceRef": "W3SVCLog1",
      "SinkRef": "W3SVCLogSink"
    }
  ]
}

Important: Add "ObjectDecoration": "instance_id={instance_id};hostname={hostname};ec2:local-hostname={ec2:local-hostname};computername={computername};env:computername={env:computername};timestamp:yyyyMMdd={timestamp:yyyyMMdd}" to your sink to be sure that EC2 metadata is appended to each log line. Kinesis Tap Agent appends the following parameters as EC2 metadata: instance_id, hostname, ec2:local-hostname, computername, env:computername, and timestamp:yyyyMMdd. If you don't want to display all of these parameters, then specify the parameters that you want to append.

For more information about configuration options, see Configuring Amazon Kinesis Agent for Microsoft Windows.

5.    Configure and start Kinesis Agent for Windows to start Kinesis Tap Agent. The following output should appear in each log line:

{
  "EventId": 7036,
  "Description": "The WinHTTP Web Proxy Auto-Discovery Service service entered the stopped state.",
  "LevelDisplayName": "Informational",
  "LogName": "System",
  "MachineName": "EC2AMAZ-GLL60A7",
  "ProviderName": "Service Control Manager",
  "TimeCreated": "2020-04-20T06:02:51.5847181Z",
  "Index": 34427,
  "UserName": null,
  "Keywords": "Classic",
  "instance_id": "i-0183xxxxxxxxxx4b7",
  "hostname": "ip-x-x-x-x.ec2.internal",
  "ec2:local-hostname": "ip-x-x-x-x.ec2.internal",
  "computername": "EC2AMAZ-GLL60A7",
  "env:computername": "EC2AMAZ-GLL60A7",
  "timestamp:yyyyMMdd": "20200420"
}

Related information

Collect, parse, transform, and stream Windows events, logs, and metrics using Amazon Kinesis Agent for Microsoft Windows

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago