How can I resolve the "Volume vol-XXXXXXXXXX cannot be modified in modification state OPTIMIZING" error in CloudFormation when I try to update my Amazon EBS volume type (AWS::EC2::Volume)?

Short description

If you use credentials when your CloudFormation stack modifies an EBS volume, then the timeout period for the volume is approximately seven hours. If you run the stack using an AWS Identity and Access Management (IAM) service role instead, then CloudFormation increases the timeout period to 36 hours.

Note: The volume's timeout period applies only to volume modification using CloudFormation. The timeout period varies for different types of resources and actions.

Typically, a fully used 1-tebibyte (TiB) EBS volume takes about six hours to migrate to a new performance configuration. For more information see, Monitor the progress of volume modifications.

Important: If you already initiated a stack update with CloudFormation, you must wait about seven hours until the stack stabilizes before completing the following steps. Your stack is stabilized when it enters UPDATE_ROLLBACK_FAILED status.

Resolution

Note: If you receive errors when running AWS Command Line Interface (AWS CLI) commands, make sure that you’re using the most recent AWS CLI version.

Recover your stack after waiting for the stack to stabilize

Consider the following example. If you change your EBS volume type from st1 to gp3, then your stack can fail to update the volume type because of volume size. In this scenario, you can recover your stack by performing the continue-update-rollback operation on your stack. For the recovery operation, you must skip the logical ID of the resource that failed to update. Your stack ends up in UPDATE_ROLLBACK_COMPLETE status.

Recover your stack using either the AWS CLI or CloudFormation console.

AWS CLI:

aws cloudformation continue-update-rollback --stack-name your-stack-name --resources-to-skip logical-ID-of-EBS-volume

CloudFormation console:

1.    Open the CloudFormation console.

2.    From the navigation pane, choose Stacks.

3.    From the Stack name column, select the stack that's stuck in UPDATE_ROLLBACK_FAILED status.

4.    Choose Stack actions, and then choose Continue update rollback.

5.    In the Continue update rollback dialog box, expand Advanced troubleshooting.

6.    In the Resources to skip - optional section, select the resources that you want to skip.

Note: Be sure to select the logical ID of the volume that failed to update.

7.    Choose Continue update rollback.

Update your stack to match the current status of your EBS volume

You must update your stack to match the current status of your EBS volume. Your CloudFormation template is updated by your stack update, and your stack changes to UPDATE_COMPLETE status.

For example, you can modify the volume type (AWS::EC2::Volume) from st1 to gp3. However, the volume type in CloudFormation remains st1 because of the rollback operation, even if the update to gp3 is accurately reflected in Amazon EBS.

1.    To monitor the status of your EBS volume, use the Amazon EC2 console or run the following command:

aws ec2 describe-volumes-modifications --volume-ids your-volume-ID

2.    After the volume type changes to gp3 in Amazon EBS, update the CloudFormation stack with the volume type (AWS::EC2::Volume) as gp3.

Note: You can modify your CloudFormation template by changing the volume type to gp3. Then, perform the update operation.

3.    Use drift detection to verify that your EBS volume resource (AWS::EC2::Volume) is in sync with your stack.

Important: Your stack update can take longer than 36 hours if you don't use a service role or if the volume size is high (~8 TiB). In such cases, your stack update could fail even after using a service role with 36 hours of stabilization. Then, you must wait until your stack stabilizes. During this period, you can't modify any other resource type. To resolve this issue, complete the following steps:

1.    Use a Retain deletion policy to remove the EBS volume from your stack without deleting the volume.

2.    Update your EBS volume outside of CloudFormation by using the Amazon EBS console instead.

3.    Import your EBS volume back into your stack.

If you want to keep the volume in the stack at all times, then you must keep the volume size below a threshold. For example, if 8 TiB of data takes 64 hours, then you can estimate the maximum size for 36 hours and lower the data size to 4 TiB. This threshold adjustment allows your update to complete.

---