write dynamic frame on s3 with xml format

Question

captionI want to write my dynamicframe on s3 with xml format . I want to get one xml file of my df. I TRIED THESES lines but got different parts of xml files . Who can help me to fix this problem please???

datasink4 = glueContext.write_dynamic_frame.from_options(frame = df, connection_type = "s3", connection_options = {"path": "s3://datalaketest1/temp"}, format = "xml", transformation_ctx = "datasink4")

score 0 · Answer 1 · answered Dec 27 '19 at 14:26

0

In this case, you need to repartition to 1, so it will create only 1 output file:

repartitioned_df = df.repartition(1)
datasink4 = glueContext.write_dynamic_frame.from_options(frame = repartitioned_df, connection_type = "s3", connection_options = {"path": "s3://datalaketest1/temp"}, format = "xml", transformation_ctx = "datasink4")

answered Dec 27 '19 at 14:26

Gokhan Atil

9,278
2
11
24

got error **An error occurred while calling o109.pyWriteDynamicFrame. Traceback (most recent call last):** – ceo Dec 27 '19 at 15:39
1- Are you sure that the line you shared was working and producing multiple files? 2- Can you verify that the frame name in the second line is correct? 3- Can you share the full error stack? – Gokhan Atil Dec 27 '19 at 15:49
yep it created different files before. – ceo Dec 27 '19 at 15:55
Thank you . I added a caption in my original post – ceo Dec 27 '19 at 15:56
have you tried coalesce(1) instead of repartition(1) ? – marie20 Apr 05 '23 at 10:32

write dynamic frame on s3 with xml format

1 Answers1