3

I am trying to read data from athena into python's pandas dataframe. However, I encounter this error

WaiterError: Waiter BucketExists failed: Max attempts exceeded. Previously accepted state: Matched expected HTTP status code: 404

Do anyone have the same problem when using data wrangler? This is my code below

import awswrangler as wr
import pandas as pd
wr.athena.read_sql_query('select * from ath_bi_orders limit 10', database='default')
user16934888
  • 33
  • 1
  • 3

4 Answers4

10

I have faced the same issue and resolved it by specifying AWS_DEFAULT_REGION env variable.

Like this.

os.environ['AWS_DEFAULT_REGION'] = 'ap-northeast-1' # specify your AWS region.

Execute it before you throw the query.

ken
  • 648
  • 2
  • 6
  • 16
4

If you have the AWS CLI installed you can also set the region with aws configure:

> aws configure
AWS Access Key ID [********************]:
AWS Secret Access Key [********************]:
Default region name [None]: us-east-1
Default output format [json]:

This fixed the error for me when I was using AWS datawrangler to run an Athena query.

abk
  • 309
  • 6
  • 15
0

You need to specify the region.

You can do it as @ken did using env var or by setting the default boto3 session like that:

boto3.setup_default_session(region_name=region)

The read_sql_query function uses the default boto3 session if no other session is provided (as mentioned here)

HagaiA
  • 193
  • 3
  • 15
0

For those who did specify a region in their session or client and are still getting this error. In the wrangler code, it looks like this error is happening because the code is trying to find an s3 bucket to write to - either passed in by the caller or defined by the workgroup.

https://github.com/aws/aws-sdk-pandas/blob/main/awswrangler/athena/_utils.py#L75

If it doesn't find the bucket, it attempts to create it with a specific name.

https://github.com/aws/aws-sdk-pandas/blob/main/awswrangler/athena/_utils.py#L445

If the bucket already exists, it will just silently exit, but if it does not exist it will attempt to create it with the name bucket_name = f"aws-athena-query-results-{account_id}-{region_name}" and wait for bucket creation to succeed.

It seems like if the role you're using does not have permissions to create it, you'll just be stuck in this waiting loop. I'm not sure why it doesnt surface as a permission error but in my experience it doesn't.

I fixed this issue by running as Admin so that the bucket could get created in the account. Once I verified the s3 bucket was created, I downgraded back to a lower perm role and was able to run my code without getting this issue.

Jeremy Fisher
  • 2,510
  • 7
  • 30
  • 59