How can I resolve API throttling or "Rate Exceeded" errors in Elastic Beanstalk?

4 minute read
0

How can I resolve API throttling or "Rate exceeded" errors when I use AWS Elastic Beanstalk?

Short description

API calls to any AWS service can't exceed the maximum allowed API request rate per second. The limit is shared across all resources per account and per AWS Region.

It doesn't matter if calls come from an application, the AWS Command Line Interface (AWS CLI), or the AWS Management Console. If API requests exceed the maximum rate per second, you receive a "Rate Exceeded" error, and API calls are then throttled. Some API calls can be made dozens of times per second, while others are limited to a few allowed calls per second.

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent AWS CLI version.

You can receive errors from API calls that are made directly to Elastic Beanstalk, as well as other AWS services that are managed by Elastic Beanstalk, such as AWS CloudFormation, Amazon Elastic Compute Cloud (Amazon EC2), Auto Scaling, and Load Balancing. The risk for throttles grows as usage in the account grows or as you add more resources to the same account.

Note: The limit of calls can vary by time of day, as our services adjust in response to load. It's best practice to dynamically adjust the rate you use the API to adjust to dynamic limit behavior.

Resolution

To prevent or resolve "Rate Exceeded" errors and throttling, try these solutions:

Find the source of high API calls

1.    In your Elastic Beanstalk event stream, identify the throttling error. Note the timeframe when you received the error. Or, if API calls are coming from an application or script, look for the timeframe in your application logs.

2.    For requests that you found in the timeframe with RequestLimitExceeded errors, use AWS CloudTrail to view events and validate the eventName, eventSource (the service), and userAgent. Match the timestamp of the error in Elastic Beanstalk events or in your logs with the errors found in CloudTrail. This will help you know what source in your account is consuming the most API calls.

Note: Manually counting CloudTrail records can be difficult. You can use Athena queries over CloudTrail, as well.

Amazon CloudWatch usage metrics can help you monitor your API usage over time. Note that not all services and API calls are supported in usage metrics at this time.

Third-party applications can make continuous calls to Elastic Beanstalk or other AWS services managed by Elastic Beanstalk. If you grant a third-party application the right to make API calls in your account, make sure that you monitor these as well.

Use best practices to reduce API usage

Use error retries, exponential backoffs, and jitter to help limit the rate of API calls. While each AWS SDK implements automatic retry logic and exponential backoff algorithms, you might have to adjust SDK configuration settings to fit your needs if the default retry logic isn't enough.

Note: The SDK settings are just one part to consider. Your own code should also use backoff, retry, and jitter logic when calling the SDK.

If you have a custom script making API calls every second, consider if this is necessary. For advance use cases, consider making a caching layer to reduce API consumption. You can also consider a multi-account strategy. It's best practice to keep a single AWS account from growing too large. Separating development or testing resources from production resources can keep development resources from taking API usage away from production resources.

Request a limit increase to your API call rate limit

If needed, you can request a limit increase to your API call rate limit. These limits are more difficult to increase than typical resource-based limits and require a strong use-case justification. Our teams will review your APIs and make sure that best practices are followed. Requests should be made well in advance of any need.

When requesting an increase, include the following information:

  • Your AWS Region and the timeframe related to the throttling issues
  • The API call that you use and call rate you need
  • A detailed use-case justification, such as the business need and technical need for the increase

Be sure to attempt using error retires, exponential backoffs, and jitter prior to sending the request. Include the results of these attempts in the request and any information related to the attempts.


Related information

Error retries and exponential backoff in AWS

Exponential Backoff and Jitter

Logging Elastic Beanstalk API calls with AWS CloudTrail

CloudWatch usage metrics

Benefits of using multiple AWS accounts

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago