How do I troubleshoot connectivity issues with my Amazon EMR notebook?

3 minute read
0

I want to troubleshoot connectivity issues with my Amazon EMR notebook.

Short description

When connecting to your Amazon EMR notebook, you might receive errors similar to the following:

  • Workspace (notebook) is stopped. Internal Error.
  • Workspace (notebook) can now be used in local mode.
  • Notebook is stopped. Service Role does not have the required permissions.
  • Notebook is stopped. Notebook security group sg-xxxxxxxx does not have an egress rule to connect with the master security group sg-yyyyyy. Please fix the security group or use the default option.
  • Notebook is stopped. Notebook security group sg-xxxxxxx should not have any ingress rules. Please fix the security group or use the default option.

Resolution

Check the service role in the EMR notebooks

1.    Validate that the notebook's AWS Identity and Access Management (IAM) role has the minimum required permissions. For more information, see Service role for EMR notebooks.

2.    Verify that the notebook has all of the permissions contained in AmazonElasticMapReduceEditorsRole. Use the AWS managed policy S3FullAccessPolicy for full access to Amazon Simple Storage Service (Amazon S3). For more information, see AWS managed policy: Amazon S3FullAccess.

3.    Remove any permissions restrictions from the bucket policy attached to the S3 bucket where the notebook is located.

Check the security groups in the EMR notebooks

1.    Validate that the security group used for your notebook has at least the minimal rules required. For more information, see Specifying EC2 security groups for EMR notebooks.

2.    It's a best practice to use different security groups for the EMR cluster and the EMR notebook. The security groups for the notebook and the cluster have different inbound and outbound rule requirements.

The notebook security group ElasticMapReduceEditors-Editor has an egress rule that allows connection to the master security group ElasticMapReduceEditors-Livy. This connection uses tcp/18888. Remove any outbound rules added in the notebook security group ElasticMapReduceEditors-Editor routing to 0.0.0.0/0.

The master security group ElasticMapReduceEditors-Livy has an ingress rule that allows connection to the notebook security group ElasticMapReduceEditors-Editor. This connection uses tcp/18888. Remove any ingress rules added in the master security group ElasticMapReduceEditors-Livyrouting to 0.0.0.0/0.

EMR cluster requirements

1.    Verify that the attached cluster is compatible and meets all cluster requirements.

2.    When Livy impersonation is turned on, verify that hadoop-httpfs is running on the EMR cluster master node.

Use the following command to check the status of hadoop-httpfs:

$ sudo systemctl status hadoop-httpfs

Use the following command to turn on hadoop-httpfs:

$ sudo systemctl start hadoop-httpfs

AWS OFFICIAL
AWS OFFICIALUpdated a year ago