How do I resolve the error "Failed to receive X resource signal(s) within the specified duration" in AWS CloudFormation?

5 minute read
0

I receive the following error message: "Failed to receive X resource signal(s) within the specified duration" for Linux Amazon Elastic Compute Cloud (Amazon EC2) instances in AWS CloudFormation.

Short description

You get this error when CloudFormation doesn't receive success signals for resources that have a CreationPolicy attribute specified with a ResourceSignal in it. The error might occur for an Amazon EC2 instance, Auto Scaling group, or a wait condition.

Note: The following resolution applies only to CloudFormation stacks that you create with Linux instances. For Windows instances, see How do I resolve the error "Failed to receive X resource signal(s) within the specified duration" for EC2 Windows instances in AWS CloudFormation?"

Resolution

Based on your use case, use the following troubleshooting steps to resolve your issue. 

Note: To prevent a stack rollback, choose Preserve successfully provisioned resources for Stack failure options in the CloudFormation console. This option means that there's no failure rollback, and the instance won't be terminated until you delete the stack.

The cfn-signal script isn't installed on the instance of the CloudFormation stack

To confirm that the cfn-signal script is installed on the instance that's configured to send signals to the stack, complete the following steps:

  1. Use SSH to connect to your Linux instance.

  2. Use of one of the following scripts to confirm that the cfn-signal script is installed.

    To confirm that the cfn-signal script is located in your directory, run the following command:

    $ sudo find / -name cfn-signal
    /opt/aws/bin/cfn-signal
    /opt/aws/apitools/cfn-init-1.4-30.amzn2/bin/cfn-sign

    To confirm that the CloudFormation helper scripts package that contains the cfn-signal script is installed, run the following command:

    $ sudo rpm -q aws-cfn-bootstrap
    aws-cfn-bootstrap-1.4-30.amzn2.noarch

    Important: The preceding command works only on distributions that use the RPM Package Manager. By default, CloudFormation helper scripts are installed on Amazon Linux Amazon Machine Images (AMIs). To install the helper scripts, see CloudFormation helper scripts reference.

There are syntax errors or incorrect values in the CloudFormation template

To find the errors and incorrect values, complete the following steps:

  1. In a code editor, open the template for your stack. Then, find the UserData property section.
  2. Check for errors with syntax, missing spaces, misspellings, and other typos.
  3. Confirm that the values for the stack, resource, and AWS Region properties are correct.
    Note: Check the bootstrap script that's included in the UserData property. The script calls cfn-signal. Check for syntax errors or incorrect values.

If you signal within the cfn-init commands, then look for information about the signal in the cfn-init logs. To search for errors in the cloud-init or cfn-init logs, use SSH to connect to your instance. Then, use the keyword "error" or "failure" to search for detailed error or failure messages in the following logs:

/var/log/cloud-init-output.log/var/log/cloud-init.log
/var/log/cfn-init.log
/var/log/cfn-init-cmd.log
/var/log/cfn-wire.log

To parse all instances of the words "error" or "failure" in /var/log/cfn or /var/log/cloud-init files, run the following command:

grep -ni 'error\|failure' $(sudo find /var/log -name cfn\* -or -name cloud-init\*)

Note: The preceding command returns the file name, line number, and error message.

The value of the timeout property for the CreationPolicy attribute is too low

The timeout property value is defined by the CreationPolicy attribute. Confirm that the value is high enough to run tasks before the cfn-signal script sends signals to CloudFormation resources.

To check the timeout property value and compare the signaling and resource failure timestamps, complete the following steps:

  1. In a code editor, open the CloudFormation template for your stack to find the timeout property value.
    Note: The timeout property value is the maximum amount of time that CloudFormation waits for a signal before it returns an error.

  2. To get an estimate of when the cfn-signal script was activated, use SSH to connect to the instance. Then, run the following command:

    less /var/log/cfn-init.log

    The log file shows a timestamp when the SUCCESS signal is sent to CloudFormation resources.
    Example:

    2019-01-11 12:46:40,101 [DEBUG] Signaling resource EC2Instance in stack XXXX with unique ID i-045a536a3dfc8ccad and status SUCCESS
  3. Open the CloudFormation console.

  4. Choose the Events view.

  5. Choose Status reason. Expand the row for the event with the status reason "Failed to receive X resource signal(s) within the specified duration."

  6. Compare the signaling timestamp with the resource failure timestamp.
    Note: For a successful completion, the script must send the signal before the instance is created or fails to create.

The cfn-signal isn't sent from the instance

Verify that the signal that CloudFormation received came from the instance. Check the cfn wire log that's available at /var/log/cfn-wire.log. If the response isn't 200, then there might be a connectivity issue between your instance and the CloudFormation's endpoint.

When you send signals from somewhere that's not your instance, use the SignalResource API. For example, you can use an AWS Lambda function to call the SignalResource API, and then send the signal to the stack. If you get an error, then use CloudWatch Logs to check your Lambda logs to understand why the signal wasn't sent to the stack.

AWS OFFICIAL
AWS OFFICIALUpdated 10 days ago