Questions tagged [pyathena]
50 questions
0
votes
1 answer
Superset with Athena: set workgroup based on user role
I'm working at a company that used to use AWS Athena and Quicksight to run sql queries and create dashboards, but now we have to use Apache Superset to do this.
While all users was using aws console, I could get Cloud Trail logs to send to the…
CleideV
- 1
0
votes
0 answers
Receiving TypeError, when PandasCursor is imported from pyathena.pandas.cursor
I want to read an excel file into pandas from an AWS S3 bucket. Everything worked fine But when I import PandasCursor, which I need for another part of the code, I receive the following error message:
import pandas as pd
import s3fs
from pyathena…
TiTo
- 833
- 2
- 7
- 28
0
votes
2 answers
Connect to Athena without access?
I need to connect to Athena using Python.
The code used is as follows:
import pyathena
import pandas as pd
athena_conn = pyathena.connect(access_key,
secret_key,
s3_staging_dir,
region_name)
df =…
Gary
- 31
- 1
- 6
0
votes
0 answers
escape single quotes in sql query while passing as variable to python
I am trying to query athena using php. My code is as below which works.
PHP
Santosh Pillai
- 1,311
- 1
- 20
- 31
0
votes
2 answers
Pyathena SQL Query with Python condition
The following python list is given:
customer_list = [123,567,494]
Now I want to run a SQL query in which I use the list from above. How can I add the condition in (customer_list) to my query?
I tried:
my_query = """
select * from my_table
where…
Minfetli
- 303
- 3
- 12
0
votes
1 answer
How to specify file name when executing query via Athena API client (Boto3)?
I have a query string and using the start_query_execution() method, I'm right now able to run my query via Athena and get the results in the form of a CSV file in my S3 bucket.
However, the file's name is a bunch of random characters and I'd like to…
Ricardo Francois
- 752
- 7
- 24
0
votes
2 answers
Fixing error in a SHOW TABLES IN DATABASE name query
I am trying to list all the table in a database in Amazon AWS Athena via a Python script.
Here is my script:
data = {'name':['database1', 'database-name', 'database2']}
# Create DataFrame
df = pd.DataFrame(data)
for index, schema in df.iterrows():
…
ozzboy
- 2,672
- 8
- 42
- 69
0
votes
1 answer
How to access nested arrays and JSON in AWS Athena
I'm trying to process some data from s3 logs in Athena that has a complex type I cannot figure out how to work with.
I have a table with rows such as:
data
____
"[{\"k1\":\"value1\", \"key2\":\"value2\"...}]"
I'd like to treat it as (1) an array to…
Tyler Wood
- 1,947
- 2
- 19
- 29
0
votes
1 answer
Error code 422 while connecting from superset to Athena
Getting Error
422 UNPROCESSABLE ENTITY while connecting from superset to Athena (superset==1.0.1)
Using below format for connections…
VenuGupta
- 41
- 1
- 3
0
votes
0 answers
Pyathena in Jupyter Notebook Can't find aws config file
I'm using a jupyter notebook, and I'm trying to query AWS athena using PyAthena module, via the notebook.
My first line in the project is:
athena = pyathena.connect(profile_name="NAME")
which works in any other platform on my computer (iPython,…
Golden
- 407
- 2
- 12
0
votes
1 answer
Pyathena "s3_staging_dir" file - how can I get this filename to use it?
I'm using Pyathena to run basic queries:
from pyathena import connect as pyathena_connect #to distinguish from other connect methods
import pandas as pd
class AthenaDataConnection():
def __init__(self, S3_STAGING_DIR, SEP=';',…
autonopy
- 429
- 8
- 12
0
votes
1 answer
How do I handle errors and retry in PyAthena?
I have an Athena query that I run every day from my local Ubuntu machine. It runs fine most times.
def get_athena_data(**kwargs):
athena_conn = connect(aws_access_key_id = access_key, aws_secret_access_key = s_key, s3_staging_dir = path,…
autonopy
- 429
- 8
- 12
0
votes
1 answer
How to loop query in pyathena?
I am using pyathena library to query schemas and storing it in pandas dataframe. I've a list which contains atleast 30,000 items.
eg. l1 = [1,2,3,4..... 29999,30000]
Now I want to pass this list items in sql query. Since I cannot pass all 30,000…
Vishal Upadhyay
- 781
- 1
- 5
- 19
0
votes
0 answers
Error: Trying to use PyAthena to access an Athena
I'm currently trying to build a data pipeline from an AWS Athena database so my team can query information using Python. However, I'm running into an issue with insufficient permissions.
We are able to query the data in Tableau, but we wanted to…
kpdebree
- 11
- 4
0
votes
2 answers
AWS Athena Table Data Update
I have started testing out AWS Athena, and it so far looks good. One problem I am having is about the updating of data in a table.
Here is the scenario: In order to update the data for a given date in the table, I am basically emptying out the S3…
khan
- 7,005
- 15
- 48
- 70