Why does my AWS Glue job return the 403 Access Denied error?

7 minute read
0

My AWS Glue job returns the 403 Access Denied error when the job tries to read/write into an Amazon Simple Storage Service (Amazon S3) bucket.

Short description

You get an Access Denied error usually because of one of the following reasons:

  • The AWS Identity and Access Management (IAM) role doesn't have the required permissions to access the bucket.
  • The Amazon S3 bucket policies don't allow the required permissions to the IAM role.
  • The S3 bucket owner is different from the object owner.
  • The Amazon Virtual Private Cloud (Amazon VPC) endpoint policy doesn't include the required permissions to access the S3 bucket.
  • The object is encrypted by AWS Key Management Service (AWS KMS), and the AWS KMS policy doesn't grant the minimum required permissions to the IAM role for using the key.
  • The S3 bucket has Requester Pays turned on.
  • Access to the S3 bucket is restricted by the AWS Organizations service control policies.

Note: If you receive errors when running AWS CLI commands, make sure that you’re using the most recent version of the AWS CLI.

Resolution

Be sure that the IAM role has the required permissions to access the S3 bucket

The IAM role running the AWS Glue job needs access to the S3 bucket. You can grant the required permissions to the IAM role by attaching an IAM policy to the IAM role. It's also a best practice that you attach the AWSGlueServiceRole Managed Policy to the IAM role to confirm that the basic AWS Glue job permissions are provided. Additionally, create and attach a Customer Managed policy for permissions to put S3 objects while writing.

Do the following to update the IAM role's permissions to access the bucket:

1.    Open the IAM console.

2.    Open the IAM role that's associated to the AWS Glue job and requires access to the bucket.

3.    In the Permissions tab of the IAM user/role, expand each policy to view its JSON policy document.

4.    In the JSON policy documents, look for policies with the bucket's name. Then, confirm that these policies allow the correct S3 actions on the bucket.

5.    If the IAM role doesn't grant the required access to the bucket, then add a policy that grants the correct permissions. For example, the following IAM policy grants the IAM role, access to put objects (s3:PutObject) into the S3 bucket DOC-EXAMPLE-BUCKET:

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Sid": "ExampleStmt",
      "Action": "s3:PutObject",
      "Effect": "Allow",
      "Resource": [
        "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
      ]
    }
  ]
}

Be sure to replace DOC-EXAMPLE-BUCKET in the policy with the name of your S3 bucket.

Be sure that the bucket policy grants the required permissions to the IAM role

Review the bucket policy for the following:

  • Any statements that explicitly deny the IAM role's access to the bucket
  • Any missing permissions and conditions that might restrict the IAM role's access

Do the following to review and modify the bucket policy to grant the required access to the IAM role:

1.    Open the Amazon S3 console.

2.    Choose Buckets from the navigation pane.

3.    From the list of buckets, open the bucket that you want to check.

4.    Choose Permissions, and then scroll down to the Bucket policy section.

5.    Review the bucket policy for any statements that deny the role's access to the bucket.

6.    Modify the bucket policy to edit or remove any statements that deny the IAM role's access to the bucket.

For sample bucket policies, see Bucket policy examples.

Be sure that the S3 bucket owner has access to objects

By default, an S3 object is owned by the AWS account that uploaded it. This is true even when the bucket is owned by another account. If other accounts can upload objects to your bucket, then check which account owns the objects that your users/roles can't access. You can check the object owner by running the GetObjectAcl command.

If IAM users/roles from other accounts upload objects to your S3 bucket, set up S3 Object Ownership. Then, add a bucket policy that requires objects to be uploaded with the bucket-owner-full-control access control list (ACL). Adding the bucket policy automatically changes the object's owner to the bucket owner when the object is uploaded with the bucket-owner-full-control ACL. For more information, see When other AWS accounts upload objects to my Amazon S3 bucket, how can I require that they grant me full control of the objects?

Be sure that the Amazon VPC endpoint policy allows the required actions to the S3 bucket

Be sure that the VPC endpoint policy includes the required permissions to access the S3 buckets and objects when both the following conditions are true:

  • Your AWS Glue job reads or writes objects into S3.
  • The connection is routed to S3 using a VPC endpoint.

For example, the following VPC endpoint policy allows access only to the bucket DOC-EXAMPLE-BUCKET. If your bucket isn't listed as an allowed resource in the policy, then the users/roles can't access your bucket through the VPC endpoint.

{
  "Statement": [
    {
      "Sid": "Access-to-specific-bucket-only",
      "Principal": "*",
      "Action": [
        "s3:PutObject"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
    }
  ]
}

Be sure to replace DOC-EXAMPLE-BUCKET in the policy with the name of your S3 bucket.

If users/roles upload objects with an ACL, then you must update the VPC endpoint policy to grant access to the PutObjectAcl action. For example:

{
  "Statement": [
    {
      "Sid": "Access-to-specific-bucket-only",
      "Principal": "*",
      "Action": [
        "s3:PutObject",
        "s3:PutObjectAcl"
      ],
      "Effect": "Allow",
      "Resource": "arn:aws:s3:::DOC-EXAMPLE-BUCKET/*"
    }
  ]
}

Be sure that the AWS KMS key's policy allows access to the IAM role

If your extract, transform, and load (ETL) job reads or writes encrypted data into Amazon S3, then be sure of the following:

  • The IAM role's policy includes the permissions required for the AWS KMS actions.
  • The AWS KMS key's policy includes the required permissions for the IAM role.

Include the following permissions in the IAM role's policy to allow the necessary AWS KMS actions:

{
  "Version": "2012-10-17",
  "Statement": {
    "Effect": "Allow",
    "Action": [
      "kms:Decrypt",
      "kms:Encrypt",
      "kms:GenerateDataKey"
    ],
    "Resource": "arn:aws:kms:us-west-2:111122223333:key/1234abcd-12ab-34cd-56ef-1234567890ab"
  }
}

Be sure to replace the ARN in the policy with the ARN of your choice.

For more information, see Setting up encryption in AWS Glue.

Review the KMS key policy to verify if the policy allows access to the AWS Glue job's role. For more information about key policies, see Using key policies in AWS KMS.

Be sure to include the Requester Pays header if the bucket has Requester Pays turned on

If the S3 bucket has Requester Pays turned on, then all requests to the bucket from the AWS Glue job must include the Requester Pays header. AWS Glue requests to Amazon S3 don't include the Requester Pays header by default. Without this header, an API call to a Requester Pays bucket fails with an Access Denied exception. To add the Requester Pays header to an ETL script, use hadoopConfiguration().set() to include fs.s3.useRequesterPaysHeader on the GlueContext variable or the Apache Spark session variable.

For more information, see How can I access Amazon S3 Requester Pays buckets from AWS Glue, Amazon EMR, or Amazon Athena?

Be sure that the AWS Organizations service control policies allow access to S3

If you're using AWS Organizations, then review the service control policies to make sure that access to Amazon S3 is allowed. For example, the following policy explicitly denies access to Amazon S3 and results in an Access Denied error.

{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Deny",
      "Action": "s3:*",
      "Resource": "*"
    }
  ]
}

For more information, see Enabling all features in your organization.


Related information

How do I troubleshoot 403 Access Denied errors from Amazon S3?

AWS OFFICIAL
AWS OFFICIALUpdated 2 years ago