5

I am trying to build a data pipeline for a data engineering project With the help of S3, Glue, Athena, etc., I am stuck when setting up glue crawler for indexing over data. Even I set up the role according to the need, but still it's giving me the following error.

 {"service":"AWSGlue","statusCode":400,"errorCode":"AccessDeniedException","requestId":"7bd42729- 
 bc4b-4e22-af2a-553860002c64","errorMessage":"Account 834025784276 is denied 
 access.","type":"AwsServiceError"}

error I am getting while setting up Glue Crawler

My role permissions

Zaheer UD Din Baber
  • 86
  • 1
  • 8
  • 1
    Are you trying to crawl data from another account? If yes have you given proper permissions for cross account access? – Prabhakar Reddy Aug 05 '22 at 03:18
  • No, am trying to connect the Glue with my own account – Zaheer UD Din Baber Aug 05 '22 at 07:47
  • @ZaheerUDDinBaber did you manage to solve it? Having the same issue even with Administrator Access. – Evandro Pomatti Aug 14 '22 at 13:19
  • @ZaheerUDDinBaber We would also be interested in any solution you might have found. It seems there is something in our infrastructure and/or Glue code which creates this issue. After creating a whole new org account (with sub-accounts) and rolling out our infrastructure there, Glue has being blocked once again (for all sub-accounts of our new org, even though we only rolled out in one sub-account). – mikey Aug 16 '22 at 10:28
  • @mikey I was able to work around it by using another account in my Organization. I'm using Terraform and nothing changed in my script, expect that I'm assuming a role when running the thing. Not sure if it helps you. – Evandro Pomatti Aug 16 '22 at 22:25
  • @mikey, I tried all possible solutions, and also seek help from AWSRepost Support as well, nobody has any idea why its causing this problem and how to fix it. – Zaheer UD Din Baber Aug 18 '22 at 08:26

2 Answers2

1

I know this is an old question, but wanted to provide a feedback on our experience with that issue, as we didn't find much on the internet.

So we started experiencing this issue at a time that seems kind of random to us, as we were using the service for 3 months, the account having been created more than 2 years ago.

We did reach out to AWS support, which conducted several steps in investigating that issue:

  • Technical investigation for AWS Glue: all seemed fine
  • AWS Fraud evaluation: Got a negative result (no fraud detected)
  • AWS "containment score" evaluation & increase: solved the issue

Funny enough, we had an issue with another service (CodePipeline, which could not start a new CodeBuild anymore) which led to a similar solution and us finding this explanation of the containment score: https://towardsaws.com/containment-score-of-aws-3a893231e948

As a TLDR; in case link goes down:

Author had an issue with not being able to start a CodeBuild build saying "Cannot have more than 0 builds in queue for the account"

Support engineer mentioned the "containment score":

Based on the below table and score, the account has a concurrent limit is 0,

0 to 200 Concurrent limit is 0

201 and more (concurrent build limit is 1)

251 and more (concurrent build limit is 5)

301 and more (concurrent build limit is 150)

901 and more (concurrent build limit is 300)

The containment score needs to be increased to resolve this issue.

So it seems that AWS evaluates for each account and depending on their use a "containment score" allowing to set limits of use of services and maybe avoid scaling up crazy in case of a hack or something. I have no idea if this score is "per service" or "global" (probably "per service")

So if you encounter this, make sure to mention this to the support engineer as it could help resolving the issue. For us, the solution took ~5 days to be found, which is a crazy delay if you're thinking about a production environment blocking issue.

Hope this helps!

Adrien0
  • 79
  • 9
0

I had the same problem and I spent more than one day to troubleshoot, apply all the advice (endpoints, permissions, roles, lake formation troubleshooting), arriving at a point where, even if it would be working, I would not consider it as solution (working by accident is not a solution). I had the idea to change the region (from eu-west to central) and it works perfectly.

Brian Tompsett - 汤莱恩
  • 5,753
  • 72
  • 57
  • 129
Sorina Branza
  • 1