How do I resolve the "Internal Failure" error when I try to create or update a stack in CloudFormation?

5 minute read
0

I want to resolve the "Internal Failure" error in AWS CloudFormation.

Short description

If you're creating or updating your CloudFormation stack, you can receive an "Internal Failure" error when an operation on a resource fails. You can also receive this error if your stack fails to deploy.

An operation on a resource can fail in the following scenarios:

  • Your resources or properties are set to incorrect values. To resolve this issue, complete the steps in the Deploy a test stack to find the incorrect values for your resources or properties section.
  • An internal workflow failed. To resolve this issue using AWS CloudTrail, complete the steps in the Find the failed API operations in your CloudTrail event logs section.

Finally, your stack can fail to deploy if you pass incorrect values to the Outputs section of your CloudFormation template. To resolve this error, complete the steps in the Check the values in the Outputs section of your CloudFormation template section.

Note: The following steps apply to only "Internal Failure" errors that you receive when you try to create or update a stack in CloudFormation.

Resolution

Deploy a test stack to find the incorrect values for your resources or properties

To find the incorrect values for your resource properties or attributes, deploy a test stack with a CloudFormation template that includes only the failed resource.

If your test stack deploys successfully, follow the steps in the Find the failed API operations in your CloudTrail event logs section.

If your test stack deployment fails, continue to remove non-required properties and attributes from the test stack until you find the incorrect values.

In the following example scenario, you receive an "Internal Failure" error when CloudFormation tries to create an AWS::Config::ConformancePack resource with AWS Config. You receive an error because the DeliveryS3Bucket property uses incorrect syntax. The DeliveryS3Bucket property accepts only a bucket name as a value (for example: bucketname). A file path that includes the bucket name isn't an acceptable value (for example: s3://bucketname).

AWSTemplateFormatVersion: 2010-09-09
Resources:
  CloudFormationCanaryPack:
    Type: AWS::Config::ConformancePack
    Properties:
      ConformancePackName: ConformancePackName
      DeliveryS3Bucket: s3://bucketname            # Incorrect value for DeliveryS3Bucket
      TemplateS3Uri: s3://bucketname/prefix

Find the failed API operations in your CloudTrail event logs

1.    Open the CloudTrail console.

2.    In the navigation pane, choose Event history.

3.    For Time range, enter a time range to isolate the failed API call, and then choose Apply.

Tip: For the From time, enter the time when the resource entered the CREATE_IN_PROGRESS or UPDATE_IN_PROGRESS status in your CloudFormation stack. For the To time, enter the time when the API call failed.

4.    To search beyond the default display of events in Event history, use attribute filters.

Note: By default, Event history uses a Read-only filter that's set to false. The Read-only filter result shows only write events for API activity and excludes read-only events from the list of displayed events.

You can use EventName to filter by the name of the returned event. If you know the API action used to create or update a resource, then you can use an EventName filter for specific API calls only. For example, the CloudFormation stack uses the AWS Config API action PutConformancePack when it creates an AWS::Config::ConformancePack resource. That means you can filter for the PutConformancePack API only. You can use EventSource to filter by the AWS service that made the API request. That means you can scroll through a list of event sources and choose the appropriate resource used in your CloudFormation template.

5.    To identify the root cause of the failure, review the error message for the returned event.

Note: Some API operation failures require you to update your original CloudFormation template, and then perform a test deployment to confirm that the error is resolved.

Check the values in the Outputs section of your CloudFormation template

In your CloudFormation template, confirm that the values in the Outputs section don't contain syntax errors. For example, remove any trailing spaces.

If you retrieve resource attributes with dynamic references, you must confirm that the attributes are available during stack deployment. To simulate this outside of CloudFormation, do the following:

1.    Make a Create* or Update* API call to the resource type with the failed attribute (to create or modify).

2.    Make a Describe* API call to retrieve current attributes of the resource during the stack creation or update process.

The following example scenario demonstrates an internal error returned by a stack when the ReplicationInstancePrivateIpAddresses attribute of the AWS::DMS::ReplicationInstance resource is passed to Outputs.

In the following example, the instance's private IP attribute is available only after the ReplicationInstance resource has switched its status to available. If the ReplicationInstance resource isn't in the available status by the time the stack processes Outputs, CloudFormation can't retrieve the private IP attribute. Then, the deployment fails.

AWSTemplateFormatVersion: 2010-09-09
Resources:
  BasicReplicationInstance:
    Type: AWS::DMS::ReplicationInstance
    Properties:
      ReplicationInstanceClass: dms.t2.small
Outputs:
  DmsInstanceIP:
    Value: !GetAtt BasicReplicationInstance.ReplicationInstancePrivateIpAddresses

Related information

Troubleshooting AWS CloudFormation

Viewing events with CloudTrail Event history

AWS OFFICIAL
AWS OFFICIALUpdated 3 years ago