I read in data from Snowflake into AWS Glue using spark, which results having a spark dataframe called df. After that I added the following to convert it to a pandas dataframe:
df2 = df.toPandas()
However, this is causing an error in AWS Glue.
Asked
Active
Viewed 177 times
0
-
Please provide details of error, metrics of data (size, columns, datatype etc.), sample code. – Azhar Khan Aug 08 '22 at 11:24
-
so the error message is: An error occurred while calling o96.collectToPython. SQL compilation error. The data is read via df = spark.read.format(SNOWFLAKE_SOURCE_NAME).options(**snowflake_options).option("dbtable",snowflake_database+"."+snowflake_schema+"."+source_table_name).load(). – gblm Aug 08 '22 at 12:21