How do I find out what stopped my AWS OpsWorks Stacks instance?

Short description

There are two ways to stop an OpsWorks Stacks instance:

• Manually stopping an instance in the OpsWorks console or by using the OpsWorks Stacks API.
• The OpsWorks Stacks autohealing feature.

Important: OpsWorks Stacks doesn't recognize start, stop, or restart operations performed in the Amazon EC2 console. For more information, see Manually starting, stopping, and rebooting 24/7 instances.

To verify what stopped your OpsWorks Stacks instance, you can do either of the following:

Review your AWS CloudTrail for simultaneous Amazon EC2 StopInstances API calls and OpsWorks Stacks StopInstance API calls

If the two API calls are logged over the same time period, then the instance was stopped manually on the OpsWorks Stacks side. If there's an Amazon EC2 StopInstances API call logged only, then autohealing was applied to the instance.

Review your instance's Agent logs to see if the OpsWorks agent was still sending its keepalive signal when the instance stopped

If successful keepalive signals are logged when the instance stopped, then the instance was stopped manually on the OpsWorks Stacks side. If the keepalive logs are missing or there are failed signal attempts logged when the instance stopped, then autohealing was applied.

If autohealing was applied to your instance, see How do I stop AWS OpsWorks Stacks from unexpectedly restarting healthy instances? If your instance was stopped manually, review the AWS Identity and Access Management (IAM) role that made the StopInstance API call. Then, determine who has access to that role and find out why they stopped the instance.

Resolution

Review your instance's CloudTrail logs for Amazon EC2 StopInstances API calls

1.    Open the CloudTrail console.

Important: Make sure that the AWS Region selected is the same Region that your instance is in.

2.    In the left navigation pane, choose Event history.

3.    In the upper left of the Event history page, select the filter dropdown list. Then, choose Resource name.

4.    In the search text box to the right of the filter dropdown list, enter your Amazon EC2 instance ID. Results for all of the events associated with the instance appear.

5.    In the Event name column, look for StopInstances.

6.    In the Event time column of the StopInstances event row, note the API call's timestamp. You will reference the timestamp when reviewing your instance's CloudTrail logs for OpsWorks Stacks StopInstance API calls.

7.    Open the event record by choosing the name of the event (StopInstances) in the Event name column.

8.    In the Event record pane, look for the "invokedBy" value. If the instance was stopped on the OpsWorks Stacks side—either manually or through autohealing—then the Amazon EC2 StopInstances API response shows the following output:

"invokedBy": "opsworks.amazonaws.com"

Note: There is no indicator in the Event record if autohealing was applied to the instance or not.

Review your instance's CloudTrail logs for OpsWorks Stacks StopInstance API calls

1.    Open the CloudTrail console.

Important: Make sure that the AWS Region selected is the same Region that your OpsWorks Stacks API endpoint is in.

2.    In the left navigation pane, choose Event history.

3.    In the upper left of the Event history page, select the filter dropdown list. Then, choose Resource name.

4.    In the search text box to the right of the filter dropdown list, enter your OpsWorks Stacks instance ID. Results for all of the events associated with the instance appear.

5.    In the Event name column, look for StopInstance.

6.    In the Event time column of the StopInstance event row, verify if the event's timestamp is the same as the Amazon EC2 StopInstances event's timestamp or not.

If the StopInstance API call is logged at the same time as the StopInstances API call, then the instance was stopped manually on the OpsWorks Stacks side.

If no StopInstance API call is logged at the same time as the StopInstances API call, then autohealing was applied to the instance.

(Optional) Review your instance's Agent logs to see if the OpsWorks agent was still sending its keepalive signal when the instance stopped

Connect to your Linux instance by using SSH (Secure Shell), or connect to your Windows instance by using the Windows remote desktop protocol (RDP). Then, check for the log file opsworks-agent.keep_alive.log in the instance's OpsWorks Agent log.

If successful keepalive signals are logged when the instance stopped, then the instance was stopped manually on the OpsWorks Stacks side. If the keepalive logs are missing or there are failed signal attempts logged when the instance stopped, then autohealing was applied.

Related information

How to set up AWS OpsWorks Stacks autohealing notifications in Amazon CloudWatch Events

How do I stop AWS OpsWorks Stacks from unexpectedly restarting healthy instances?

How do I troubleshoot "Internal Error" messages when stopping an AWS OpsWorks Stacks instance that's in the "stop_failed" state?

---