How can I rotate a Secrets Manager secret in a private VPC?

4 minute read
0

I tried to rotate an AWS Secrets Manager secret for an AWS service in an Amazon Virtual Private Cloud (Amazon VPC). However, the operation failed and Amazon CloudWatch Logs show that the AWS Lambda task timed out.

Short description

Secrets Manager can't rotate secrets for AWS services running in Amazon VPC private subnets because these subnets don't have internet access.

Resolution

Follow these instructions to configure an Amazon VPC interface endpoint to access your Secrets Manager Lambda function and private Amazon Relational Database Service (Amazon RDS) instance. In the following example, a private Aurora RDS instance in Amazon VPC named vpc-0abb11f5a28a8abe7 is used.

Important:

Create SGs for the Secrets Manager VPC endpoint, RDS instance, and Lambda rotation function

Follow these instructions for creating security groups (SGs) using the AWS CLI.

1.    SG for Secrets Manager Amazon VPC endpoint:

Note: Replace vpc-id vpc-0abb11f5a28a8abe7 with your VPC ID.

$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name SMVPCEndpointSG --description "secretsmanager VPCEndpoint SG"
{
    "GroupId": "sg-vpc-endpoint"
}

2.    Security group for Lambda rotation function:

$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name LambdaFunctionSG --description "Lambda Rotation Function SG"
{
    "GroupId": "sg-lambda-function"
}

3.    (Optional) Create an SG for the RDS instance:

Note: This step is required if your RDS instance uses only the default security group.

$ aws ec2 create-security-group --vpc-id vpc-0abb11f5a28a8abe7 --group-name RDSInstanceSG --description "RDS Instance SG"
{
    "GroupId": "sg-rds-instance"
}

Add rules to Amazon VPC endpoint and RDS instance SGs

1.    Get the CIDR range for your VPC:

$ aws ec2 describe-vpcs --vpc-ids vpc-0a05c93c7ef7a8a1c --query 'Vpcs[].CidrBlock' --output text
10.0.0.0/16

2.    Configure the security group rules for the Amazon VPC endpoint to allow inbound traffic on port 443 from your VPC: 

$ aws ec2 authorize-security-group-ingress --group-id sg-vpc-endpoint --protocol tcp --port 443 --cidr 10.0.0.0/16

3.    Configure the RDS instance SG to allow inbound connections from the Lambda function security group:

Note:

  • Replace your-rds-security-group with your SG (either an existing SG or the optional RDS instance SG).
  • Replace your-db-port with the port that your database is configured to use.
$ aws ec2 authorize-security-group-ingress --group-id your-rds-security-group --protocol tcp --port your-db-port --source-group sg-lambda-function

Attach SGs to AWS resources

1.    If you created the optional RDS instance SG, modify the RDS instance configuration:

Note: Replace your-existing-rds-security-groups with the group or groups attached to the RDS instance.

$ aws rds modify-db-instance --db-instance-identifier your-rds-instance --vpc-security-group-ids sg-rds-instance your-existing-rds-security-groups

2.    Follow the instructions to update the Lambda function configuration:

$ aws lambda update-function-configuration --function-name your-lambda-function \
--vpc-config SubnetIds=subnet-076c28105d486f3bd,subnet-0af00c796ccdc725f,SecurityGroupIds=sg-lambda-function

Create an Amazon VPC interface endpoint for the Secrets Manager service and associate it with an SG

Follow the instructions for creating an interface endpoint:

Note: Replace your-region with your AWS Region and the subnet IDs used for your RDS instance.

$ aws ec2 create-vpc-endpoint --vpc-id vpc-0abb11f5a28a8abe7 --vpc-endpoint-type Interface \
--service-name com.amazonaws.your-region.secretsmanager --subnet-ids subnet-076c28105d486f3bd subnet-0af00c796ccdc725f \
--security-group-ids sg-vpc-endpoint

Important: Your Amazon VPC must have the DNS hostnames and DNS resolution attributes activated. For more information, see Viewing and updating DNS support for your VPC.

Verify that Secrets Manager can rotate the secret

1.    Follow the instructions for rotating the Secrets Manager secret:

Note: Replace your-secret with your Secrets Manager secret. 

$ aws secretsmanager rotate-secret --secret-id your-secret

Secrets Manager retries the previous rotation.

Note: Because previous attempts to rotate the password were unsuccessful, you might receive an output similar to the following:

An error occurred (InvalidRequestException) when calling the RotateSecret operation: A previous rotation isn't complete. That rotation will be reattempted.

2.    Monitor the function in the AWS Lambda console. If the rotation is successful, then the Amazon CloudWatch log streams contain an entry similar to the following:

[INFO] 2019-10-22T07:59:32.627Z 96179023-5b67-4e98-a057-885f68bc69f2 finishSecret: Successfully set AWSCURRENT stage to version 175b5e38-341f-4cd0-8c58-2b1e49769642 for secret arn:aws:secretsmanager:your-region:your-account:secret:your-secret

3.    Retrieve the Secrets Manager secret to confirm that it rotated successfully:

Note: Replace your-secret-arn with your Secrets Manager secret ARN.

aws secretsmanager get-secret-value --secret-id your-secret-arn

Note: The Secrets Manager rotation function runs asynchronously in the background. The rotation function can take several minutes to complete.

Related information

How can I rotate an AWS Secrets Manager secret for a DB connection that requires an SSL connection?

AWS OFFICIAL
AWS OFFICIALUpdated 10 months ago