Why did the running task count change in my Amazon ECS service?

6 minute read
0

The running task count in my Amazon Elastic Container Service (Amazon ECS) service has increased or decreased.

Short description

The running task count for your Amazon ECS service might increase or decrease due to multiple reasons. Some of the common causes are the following:

  • Your Amazon ECS tasks are failing health checks.
  • AWS CloudFormation or AWS Cloud Development Kit (AWS CDK) updated the desired count in the ECS service.
  • You manually updated the desired count for your ECS service.
  • The Amazon ECS Service Auto Scaling updated the number of tasks based on the scaling policy.
  • You updated the minimum or maximum count in Application Auto Scaling using the RegisterScalableTarget API.
  • The deployment configuration of your ECS service modified the running task count during deployment.

Resolution

Your ECS tasks are failing health checks

CloudFormation or AWS CDK updated the desired count in your ECS service

If you created your ECS service with CloudFormation or CDK without specifying the DesiredCount field, then the desired count is set to the default value of 1. However, when the service is updated through CloudFormation or CDK without specifying the DesiredCount field, the existing desired count in the current deployment is used for the new deployment.

To troubleshoot this issue, review AWS CloudTrail events to check if the UpdateService API call was made by CloudFormation. You can verify this same by checking that the userAgent field in the Event record section specifies the following:

"userAgent": "cloudformation.amazonaws.com"

To resolve this issue, remove the desiredCount parameter from the CloudFormation or CDK code during the update.

You manually updated the desired count for your ECS service

If you manually updated the ECS Service using the UpdateService API, Amazon ECS console, or AWS Command Line Interface (AWS CLI), then update the service again to change the desired count to an appropriate value. For more information, see Updating a service and update-service.

To troubleshoot this issue, review CloudTrail events to check whether the UpdateService API call was made from the Amazon ECS console or AWS CLI. You can view this information in the userAgent field in the CloudTrail record.

Example:

"userAgent": "console.amazonaws.com"

The Amazon ECS Service Auto Scaling updated the number of tasks based on the scaling policy

The Amazon ECS Service Auto Scaling might have increased or decreased the desired count based on the scaling policy. Review the service and check for any recent scaling activity.

To view the scaling activities, do the following:

  1. Open the Amazon ECS console.
  2. In the navigation pane, choose Clusters.
  3. Choose the cluster that you want to check.
  4. Choose the Services tab.
  5. Choose the service that you want to check.
  6. Choose the Events tab.

If there was a scaling event, then you might view informational messages, such as the following, in the Message column:

Successfully set desired count to 1. Change successfully fulfilled by ecs. Cause: monitor alarm TargetTracking-service/service-autoscaling/sample-webapp-AlarmLow-fcd80aef-5161-4890-aeb4-35dde11ff42c in state ALARM triggered policy TargetTrackingPolicy.

If the tasks are stopped due to a scaling activity, review the target tracking or step scaling policies. Then, update the desired count according to your application’s requirements.

To view the Application Auto Scaling configuration, do the following:

  1. Open the Amazon ECS console.
  2. In the navigation pane, choose Clusters.
  3. Choose the cluster that you want to check.
  4. Choose the Services tab.
  5. Choose the service that you want to check.
  6. Choose the Auto Scaling tab.
    Note: If you're using the new Amazon ECS console, choose Configuration and tasks, and then view the information under Service configuration.

You updated the minimum and maximum count in Application Auto Scaling using the RegisterScalableTarget API

If you used the RegisterScalableTarget API to update the scalable target, then check the MinCapacity property value specified in the API. If you specified a new minimum value, then Application Auto Scaling uses this value as the new minimum value that the tasks can be scaled down to.

Note: Application Auto Scaling doesn't scale down the desired count to less than the specified minimum value.

The deployment configuration of your ECS service modified the running task count during deployment

Review CloudTrail events to check if the UpdateService API was called recently. You can view the deployment configuration of your ECS service by checking the deploymentConfigration request parameter in the UpdateService API.

-or-

Review the deployment configuration of your ECS service in the Amazon ECS console by doing the following:

  1. Open the Amazon ECS console.
  2. In the navigation pane, choose Clusters.
  3. Choose the cluster that you want to check.
  4. Choose the Services tab.
  5. Choose the service that you want to check.
  6. Choose the Deployments tab.

The deployment configuration parameters control how many tasks are run during a deployment and the sequence of stopping and starting tasks. If the deployment configuration is set incorrectly, then the number of tasks running during the deployment is reduced. Therefore, it's a best practice to configure the value of the minimumHealthyPercent parameter to higher than 0%.

The minimumHealthyPercent parameter represents the lower limit on either of the following as a percentage of the desired number of tasks for the service:

  • Number of tasks that should be running for a service during a deployment
  • Number of tasks that should be running when a container instance is draining.

The value of this parameter is rounded up. For example, if minimumHealthyPercent is 50% and the desired number of tasks is four, then the scheduler can stop two existing tasks before starting two new tasks. If minimumHealthyPercent is 75% and the desired number of tasks is two, then the scheduler can't stop any tasks due to the resulting value also being two.

The maximumPercent parameter represents the upper limit on either of the following as a percentage of the desired number of tasks for a service:

  • Number of tasks that should be running for a service during a deployment
  • Number of tasks that should be running when a container instance is draining.

The value of this parameter is rounded down. For example, if maximumPercent is 200% and the desired number of tasks is four, then the scheduler can start four new tasks before stopping four existing tasks. If maximumPercent is 125% and the desired number of tasks is three, then the scheduler can't start any tasks due to the resulting value also being three.

For more information, see Rolling update.


Related information

Service auto scaling

How do I resolve the "[AWS service] was unable to place a task because no container instance met all of its requirements" error in Amazon ECS?

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago