How do I resolve my unified CloudWatch agent not pushing log events?

Short description

If your unified CloudWatch agent doesn't push log events, then the following are possible causes:

• Out of sync metadata
• Outdated version of the CloudWatch agent
• Failure to connect to the CloudWatch Logs endpoint
• Incorrect account, AWS Region, or log group configurations
• Insufficient AWS Identity and Access Management (IAM) permissions
• CloudWatch agent run errors
• Timestamp issues
• Missing or incorrect value in the state file for the log
• Incorrect source log file configuration
• PutLogEvents constraints
• High force_flush_interval value

Note: The following resolution doesn't apply to CloudWatch Container Insights. For more information on troubleshooting CloudWatch Container Insights, see Troubleshooting Container Insights and How do I troubleshoot the pod status in Amazon EKS?

Resolution

Note: If you receive errors when you run AWS Command Line Interface (AWS CLI) commands, then see Troubleshoot AWS CLI errors. Also, make sure that you're using the most recent AWS CLI version.

Out of sync metadata

If you have out of sync metadata, then the CloudWatch agent doesn't work as intended. To resolve this issue, complete the following steps:

• Make sure that you use AWS CloudFormation, AWS Systems Manager Agent (SSM Agent), user data scripts, or the AWS CLI to install the CloudWatch agent at launch.
• Make sure that you create an AMI before you install the CloudWatch agent. AMIs capture unique information from the original instance.
• For Windows instances, use Sysprep when you work with an AMI. For more information, see How can I use Sysprep to create and install custom reusable Windows AMIs?

Outdated version of the CloudWatch agent

If you have an outdated version of the CloudWatch agent, then download the CloudWatch agent package release notes and latest version number. The latest version might include updates that resolve your issue.

Failure to connect to the CloudWatch Logs endpoint

To test your connectivity to the CloudWatch Logs endpoint, use one of the following commands:

Note: Replace example-region with the required Region.

telnet logs.<example-region>.amazonaws.com 443

nc -zv logs.<example-region>.amazonaws.com 443

If you receive connection failures, then complete the following steps:

• Make sure that the security group and network access control lists (network ACL) allow connectivity.
• For public endpoints, use an internet gateway or a NAT gateway for your instance.
• If you use Amazon Virtual Private Cloud (Amazon VPC) endpoints, then make sure that the endpoint resolves to an Amazon VPC IP. Also, make sure that the endpoint security group allows access from the source instance.

Incorrect account, Region, or log group configurations

For the CloudWatch agent configuration file, make sure that the specified Region matches the console Region. Also, make sure that the logs are checked in the correct account.

Optionally, use the following common-config.toml files to override system defaults for the CloudWatch agent:

Linux:

/opt/aws/amazon-cloudwatch-agent/etc/common-config.toml

/etc/amazon/amazon-cloudwatch-agent/common-config.toml

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\common-config.toml

Insufficient IAM permissions

The CloudWatch agent uses credentials from either the IAM user or IAM role policy to push log events to the CloudWatch service. Before you publish a log event, create a log group and log stream. If there's no log group or log stream, then the CloudWatch agent creates them.

Check that your policy includes the following IAM permissions:

"logs:CreateLogGroup","logs:CreateLogStream",
"logs:PutLogEvents",
"logs:DescribeLogStreams"

Add any missing IAM permissions to the user policy or the role policy.

Note: When you create IAM roles and users, it's a best practice to use the CloudWatchAgentServerPolicy and CloudWatchAgentAdminPolicy policies that are created by Amazon.

CloudWatch agent run errors

Verify that the CloudWatch agent is running. If the agent isn't running, then check the log file location for errors:

Linux:

/opt/aws/amazon-cloudwatch-agent/logs/amazon-cloudwatch-agent.log

Windows:

$Env:ProgramData\Amazon\AmazonCloudWatchAgent\Logs\amazon-cloudwatch-agent.log

Note: Logs might be specified in a custom logfile location. Check the agent configuration file to identify any custom log locations.

Use the debug parameter to turn on verbose debug logging in the configuration file. If you use the run_as_user parameter, then check that the user has permissions to the log location. If the user doesn't have the necessary permissions, then CloudWatch can't write logs to the location.

If you observe IMDS related issues when you start the CloudWatch agent, then complete the following steps:

• Check if you can access the metadata service. Check the connectivity to the metadata endpoint. For more information, see Retrieve instance metadata.
• Review your OS level routes and firewalls to check if there are any rules that block connectivity to the metadata endpoint. For Linux, see How can I troubleshoot instance metadata issues on my EC2 Linux instance? For Windows, see Why does my Amazon EC2 Windows instance generate a "Waiting for the metadata service" error?

Timestamp issues

Check for log event timestamps that are older than 14 days or more than two hours in the future. The PutLogEvents command doesn't allow log batches for either time period. Also, check that the system time service on the instance is correctly configured. For more information, see Set the time for your Linux instance or Set the time for your Windows instance.

Missing or incorrect value in the state file for the log

To resolve a missing or incorrect value in the state file for the log, complete the following steps:

1. Make sure that your state files are in the correct location:
   Linux:

   /opt/aws/amazon-cloudwatch-agent/logs/state
     
   

   Windows:

   C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\state

   Note: Make sure that the preceding state files contain the same size value as the log file. This allows new data to be read and published to CloudWatch.

2. Check the size value of the state file:
   Linux:

   sudo cat /opt/aws/amazon-cloudwatch-agent/logs/state/_var_log_httpd_access_log
     
   

   Windows:

   gc 'C:\ProgramData\Amazon\AmazonCloudWatchAgent\Logs\state\C__Program_Files_log.log'
     
   

   Note: The output of the preceding commands is a numerical value of the current offset. For example, if the output is 34, then the current offset is at the 34th byte of the log file. Logs are read at this byte when new logs are written to the log file.

3. Check the size value of the log file:
   Linux:

   stat /var/log/httpd/access_log | grep Size | awk '{print $2}'
     
   

   Windows:

   Get-Item 'C:\Program Files\log.log' | Format-List | findstr Length

4. Check that the outputs from the preceding step 2 and 3 are the same.

Incorrect source log file configuration

To resolve issues from an incorrect source log file configuration, complete the following steps:

• Check if the CloudWatch agent configuration includes the log file that you want to monitor.
• Make sure that the log file has the correct file format, and make sure that the correct names and locations are used.
• From the current configuration, check that the logs are pushed to the same log group and log stream that you want to retrieve data from.
• Review the log file that you're monitoring on your instance. Check that the source log file contains the data that you want to be published to the log group.

PutLogEvents constraints

CloudWatch agent uses the PutLogEvents API to publish log events to CloudWatch log groups. Make sure that you comply with the constraints of this API. For more information, see PutLogEvents.

High force_flush_interval value

The force_flush_interval specifies the maximum amount of time that the logs can remain in the memory buffer before they are sent to the server. If the size of the logs reaches the maximum buffer of 1 MB, then the logs are immediately published to CloudWatch. The logs are published to CloudWatch regardless of the interval set by the force_flush_interval.

If the interval is set to a high value and the log size doesn't reach 1 MB, then the logs remain in the buffer. The logs remain in the buffer until the buffer is full or the interval elapses. During this interval, the logs aren't available in CloudWatch. Make sure that you set your interval value to the correct size. It's a best practice to set your interval value based on the log size and intervals that you want them to be published at.

Related information

Troubleshooting the CloudWatch agent