0

I want to export a dataframe within DataBricks to CSV. I am using the following code:

data= data_raw.copy()
df = spark.createDataFrame(data)
display(df)

Then I am using the right click option to export the dataframe. The data file contains , seperated values. I want to change the delimiter from , to ; but I don't know how.

Do you have any ideas?

data_raw contains two columns: 1 Number 2 Text

Example Data

Number Text
1 This is a text, containing letters. It can contain multiple ,
2 This a new text,it contains several letters
Oli
  • 9,766
  • 5
  • 25
  • 46
bint
  • 3
  • 3
  • @bint, can you please provide the sample data that variable `data_raw` holds? – Saideep Arikontham Sep 26 '22 at 15:25
  • @Chris thank's for your answer. It is working for the basic export. The challenge is that i want to change the delimeter before doing the export. The standard delimeter is a comma: , . I want to export the data with a semicolon: ;. – bint Sep 26 '22 at 15:31
  • @SaideepArikontham I want to export a basic pandas dataframe. This basic pandas dataframe is being created with pd.read_csv() command. Do you need any further information? – bint Sep 26 '22 at 15:33
  • Please provide sample data stored inside `data_raw`. – Saideep Arikontham Sep 26 '22 at 15:41
  • @SaideepArikontham The dataframe data_raw is a pandas dataframe and contains two columns: 1. Column = Number 2. Column = text. Example data: 1, This a test, containing a test. – bint Sep 26 '22 at 15:44
  • Understood. Please add sample data for that (say 2 sample rows to give an idea about how the data looks like) – Saideep Arikontham Sep 26 '22 at 15:44
  • @SaideepArikontham I have created a dataframe in the post above. Thanks for your great support so far! – bint Sep 26 '22 at 15:47
  • @bint, how many rows are you dealing with here? – Saideep Arikontham Sep 26 '22 at 16:18
  • 6687 Rows up to 480.000 Rows but not more. – bint Sep 26 '22 at 16:27
  • @bint You can just add`.option('delimiter',';')` to your csv output code – Chris Sep 26 '22 at 16:33
  • @Chris is it possible to combine command ".option('delimeter',';') with spark.createDataFrame(data)? Should the command look like the following: spark.createDataFrame(data).option('delimeter',';') ? – bint Sep 26 '22 at 16:52
  • The delimiter is for output, not for creation of a dataframe - if you're trying to use the export button w/a custom delimiter I'm not aware of a way – Chris Sep 26 '22 at 16:57

0 Answers0