0

I have a dataframe format of

id|name|1|[aaa]|2|[bbb]

Need to write csv file in format

id,name
1,""[""aaa""]""
2,""[""bbb""]""

Not getting correct format. Could you please help. Thanks in advance.

My code like

data=[{"id":1,"name":["aaa"]},{"id":2,"name":["bbb"]}]
df_t=spark.createDataFrame(data)
df_t=df_t.withColumn("name",concat(lit('"'), col('name').cast(StringType()),lit('"')))
df_t.show()
members_df1.coalesce(1).write.mode("overwrite").options(header="True", 
delimiter=",",escape="\"" ).csv("s3://bucket/folder")
NorSer
  • 134
  • 2
  • 13
Sambit
  • 11
  • 1
  • Your desired format is incorrect. Having said that, the Python `csv` module has a wide range of ways to shoot yourself in the foot; please review the documentation. You probably will need to implement your own CSV "dialect", and of course, no well-behaving CSV application will be able to read the files you produce. – tripleee Sep 22 '22 at 08:15

0 Answers0