How do I resolve the error ‘The security token included in the request is expired’ when running Java applications on Amazon EC2?

5 minute read
1

My Java applications using the AWS SDK for Java on an Amazon Elastic Compute Cloud (Amazon EC2) instance receive an exception similar to the following: com.amazonaws.AmazonServiceException: The security token included in the request is expired (Service: AmazonSQS; Status Code: 403; Error Code: ExpiredToken; Request ID: 12a345b6-78cd-901e-fg23-45hi67890jkl) How can I resolve this?

Short description

All application API requests to Amazon Web Services (AWS) must be cryptographically signed using credentials issued by AWS.

If your application uses temporary credentials when creating an AWS client, then the credentials expire at the time interval specified during their creation. You must refresh the credentials before they expire.

Another reason for expiration is using the incorrect time. A consistent and accurate time reference is crucial for many server tasks and processes. If your instance's date and time aren't set correctly, the AWS credentials are rejected.

If your application is running on an Amazon EC2 instance, it's a best practice to use an AWS Identity and Access Management (IAM) role assigned to the instance. Using an IAM role allows the use of a default service constructor. The default constructor client searches for credentials by using the default credentials provider chain, in the following order:

  1. In system environment variables: AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.
  2. In the Java system properties: aws.accessKeyId and aws.secretKey.
  3. In the default credentials file (the location of this file varies by platform).
  4. In the instance profile credentials contained in the instance metadata associated with the IAM role for the EC2 instance. Attaching an instance profile to your instance adds instance profile credentials to the default credentials provider chain. For more information, see Using an IAM role to grant permissions to applications running on Amazon EC2 instances.

If instance profile credentials are available, then the default client constructor creates an instance of the AWS SDK InstanceProfileCredentialsProvider class. AWS uses this class to sign API requests with AWS credentials using temporary security credentials from Amazon EC2 instance metadata.

Important: If your application uses the AWS SDK ProfileCredentialsProvider class to provide temporary AWS credentials, then you're responsible for checking for and refreshing credentials before they expire. Not checking or refreshing your credentials increases the likelihood of application failures caused by ExpiredToken errors.

Resolution

Use Amazon Time Sync Service or NTP sources

Configure the Amazon Time Sync Service, or another Network Time Protocol (NTP) source on your Amazon EC2 instance. This makes sure that your Linux instance has a consistent and accurate time reference. For more information, see Set the time for your Linux instance, or Set the time for a Windows instance.

Use custom temporary AWS credentials

Refresh temporary credentials five minutes before their expiration.

Use an IAM role assigned to an instance

Attach an instance profile to your instance. For more information, see Using an IAM role to grant permissions to applications running on Amazon EC2 instances. Verify that no other credentials are specified in your code or on the instance. The instance profile credentials are the last place the default credential provider chain searches for credentials. If you have credentials located anywhere earlier in the search chain, then those credentials prevent the IAM from being used. For more information, see Working with AWS Credentials.

To see the AWS credentials for an IAM role that's attached to an instance, run the following commands from a Linux shell or from Windows PowerShell (v3.0 or later). Be sure to replace examplerole with the name of your IAM role.

Linux

Use the curl command to see AWS credentials:

$ curl http://169.254.169.254/latest/meta-data/iam/security-credentials/examplerole

This command returns output similar to the following:

{
    "Code" : "Success",
    "LastUpdated" : "2016-04-26T16:39:16Z",
    "Type" : "AWS-HMAC",
    "AccessKeyId" : "AKIAIOSFODNN7EXAMPLE",
    "SecretAccessKey" : "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
    "Token" : "token",
    "Expiration" : "2016-04-27T22:39:16Z"
}

Note: If running the preceding curl command returns a 404 error, check the following:

1.    Use the following command to verify that HTTP proxy is disabled for the metadata IP address:

$ export NO_PROXY=169.254.169.254

2.    Verify that the instance isn't making multiple concurrent requests and running multiple sessions in parallel. Multiple concurrent requests and multiple sessions running in parallel might cause throttling by the Instance Metadata Service (IMDS). To mitigate this, use caching and retries with exponential backoff. As with any service, calls might fail occasionally. Clients are expected to retry when this occurs. For more information, see Query throttling.

To implement retries, modify AWS_METADATA_SERVICE_NUM_ATTEMPTS. You can set options using environment variables, in the ~/.aws/config file, or in the user's botocore session. For more information, see Configuration in the Boto3 DOCS 1.17.6 documentation.

Example:

AWS_METADATA_SERVICE_TIMEOUT = 10
AWS_METADATA_SERVICE_NUM_ATTEMPTS = 5

3.    If you're running the curl test in a docker container, adjust the instance-metadata-options http-put-response-hop-limit:

$ aws ec2 modify-instance-metadata-options --instance-id $(curl 169.254.169.254/latest/meta-data/instance-id) --http-put-response-hop-limit 2 --http-endpoint enabled

For more information, see Add defense in depth against open firewalls, reverse proxies, and SSRF vulnerabilities with enhancements to the EC2 Instance Metadata Service.

4.    Verify that the instance profile is properly attached to the instance.

Windows

Use the Invoke-RestMethod command to see AWS credentials:

PS C:\> Invoke-RestMethod http://169.254.169.254/latest/meta-data/iam/security-credentials/examplerole

This command returns output similar to the following:

Code            : Success
LastUpdated     : 2016-07-18T18:09:47Z
Type            : AWS-HMAC
AccessKeyId     : AKIAIOSFODNN7EXAMPLE
SecretAccessKey : wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Token           : token
Expiration      : 2016-04-27T22:39:16Z

Use these commands to check the latest temporary credentials for the instance. These credentials automatically rotate or refresh approximately five minutes before the expiration of the assigned temporary credentials.


Related information

Working with AWS credentials (AWS SDK for Java)

Using credentials (AWS SDK for Java 2.0)

IAM roles for Amazon EC2

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago