0

Is it always necessary to convert glue dynamic frame to spark dataframe before writing to snowflake? I didn't find any other way anywhere. This conversion for 20 million records is taking most of the time. Writing only takes 2 mins.

Has anyone done this like writing the dynamic frame directly to snowflake? Also learnt AWS Glue doesn't support JDBC connect to Snowflake. So giving connect details in job parameters.

Mark Rotteveel
  • 100,966
  • 191
  • 140
  • 197
Amlan Mishra
  • 1
  • Glue doesn't support snowflake directly . are you using glue native transformations ? if yes did you try writing to s3 first and then load it from there? If no then just read to spark dataframe and write with out dynamicframe in bewtween ? – Prabhakar Reddy Oct 11 '20 at 07:09
  • Hi Prabhakar, I am reading from glue data catalog to glue dynamic frame. Doing some glue native transformation. Converting to spark dataframe. Doing some spark transformation. Is it possible to load data from glue data catalog to spark dataframe directly? – Amlan Mishra Oct 11 '20 at 07:12
  • you should try https://docs.aws.amazon.com/glue/latest/dg/aws-glue-programming-etl-glue-data-catalog-hive.html spark-sql to load from catalog into dataframe – Prabhakar Reddy Oct 11 '20 at 07:16

0 Answers0