How can I correctly update my Auto Scaling group when I update my AWS CloudFormation stack?

3 minute read
2

I made updates to my Auto Scaling group, but I didn't get the results that I expected during an AWS CloudFormation stack update.

Short description

The AWS::AutoScaling::AutoScalingGroup resource uses the UpdatePolicy attribute to define how an Auto Scaling group resource is updated when the AWS CloudFormation stack is updated. If you don't have the correct settings configured for the UpdatePolicy attribute, your rolling update can produce unexpected results.

You can use the AutoScalingRollingUpdate policy to control how AWS CloudFormation handles rolling updates for an Auto Scaling group. This common approach keeps the same Auto Scaling group, and then replaces the old instances based on the parameters that you set.

Note: The following resolution focuses on the AutoScalingRollingUpdate policy only. You can use the AutoScalingReplacingUpdate policy to immediately roll back the stack without failure.

The AutoScalingRollingUpdate policy supports the following configuration options:

"UpdatePolicy": {
  "AutoScalingRollingUpdate": {
    "MaxBatchSize": Integer,
    "MinInstancesInService": Integer,
    "MinSuccessfulInstancesPercent": Integer,
    "PauseTime": String,
    "SuspendProcesses": [ List of processes ],
    "WaitOnResourceSignals": Boolean
  }
}

Resolution

To find out what's preventing your Auto Scaling group from updating correctly during a stack update, work through the following troubleshooting scenarios as needed:

Configure WaitOnResourceSignals and PauseTime to avoid problems with success signals

In your AutoScalingRollingUpdate policy, set the WaitOnResourceSignals property to false.

Note: If WaitOnResourceSignals is set to true, then PauseTime changes to a timeout value. AWS CloudFormation waits to receive a success signal until the maximum time specified by the PauseTime value. If a signal isn't received, AWS CloudFormation cancels the update. Then, AWS CloudFormation rolls back the stack with the same settings, including the same PauseTime value. For more information, see Add an UpdatePolicy to an Auto Scaling Group.

Configure MinSuccessfulInstancesPercent to avoid stack rollback

If you're replacing a large number of instances during a rolling update and are waiting for a success signal for each instance, then set the value of the MinSuccessfulInstancesPercent property in your AutoScalingRollingUpdate policy.

Note: Setting theMinSuccessfulInstancesPercent property prevents AWS CloudFormation from rolling back the entire stack if only a single instance fails to launch. For more information, see UpdatePolicy Attribute.

Configure SuspendProcesses to avoid unexpected changes to the Auto Scaling group

During a rolling update, suspend the following Auto Scaling processes:

  • HealthCheck
  • ReplaceUnhealthy
  • AZRebalance
  • AlarmNotification
  • ScheduledActions

Important: If you're using your Auto Scaling group with Elastic Load Balancing, don't suspend the following processes: Launch, Terminate, and AddToLoadBalancer. These processes are required to make rolling updates.

Note: If an unexpected scaling action changes the state of the Auto Scaling group during a rolling update, the update can fail. The failure can result from an inconsistent view of the Auto Scaling group by AWS CloudFormation.


AWS OFFICIAL
AWS OFFICIALUpdated 4 years ago
2 Comments

If you specify the MinSuccessfulInstancesPercent property, you must also enable the WaitOnResourceSignals and PauseTime properties.

https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-attribute-updatepolicy.html#cfn-attributes-updatepolicy-rollingupdate-minsuccessfulinstancespercent

AWS
replied 5 months ago

Thank you for your comment. We'll review and update the Knowledge Center article as needed.

profile pictureAWS
MODERATOR
replied 5 months ago