How to load a spark dataframe data into synapse Dedicated pool in Synapse?

Question

I have a dataframe having 100000 records that we have got after doing transformations, now i have to load all this data into synapse dedicated Pool in COUNTRY_TABLE. How can I achieve this in Synapse?

Few Other Queries

Is it Compulsory to create a schema of our dataframe columns in dedicated pool table?
How we can Overwrite the data Everytime in dedicated Pool from using query in spark notebook, if new data came i want to overwrite old data with the new data everytime.

I have created a Schema also for my destinaton table in dedicated pool with all column names we have in our spark dataframe

Aravind Yarram · Answer 1 · 2022-11-29T18:11:46.930

0

You can use the Azure Synapse Dedicated SQL Pool Connector for Apache Spark to load data to Synapse Spark Pool as the number of records is relatively low. Another option is to use the COPY command.

edited Nov 29 '22 at 18:11

answered Nov 29 '22 at 17:07

Aravind Yarram

78,777
46
231
327

is there any other way of doing this – BigData Lover Nov 29 '22 at 17:08
what about if records are higher?? – BigData Lover Nov 29 '22 at 17:09
hi @Aravind, we are not copying data from external files, we have created the table in spark pool only by using Query spark.sql("CREATE DATABASE IF NOT EXISTS DUMMY_DB") df2.write.mode("overwrite").saveAsTable("DUMMY_DB.RESULT") – BigData Lover Nov 30 '22 at 07:12
it will store data in spark pool db in table Result. Now this data i want to load into SQL pool everytime by overwrite the existing data – BigData Lover Nov 30 '22 at 07:13

How to load a spark dataframe data into synapse Dedicated pool in Synapse?

1 Answers1

Linked