0

I have a csv file which looks like this-

enter image description here

I've read it and stored it in a dataframe as follows:

query_df=spark.read.csv('./rules/rules.csv',header=True)
query_df.show(truncate=False)

However when I view this dataframe using the .show() method above, a part of the last value under the 'Query' column is getting omitted:

enter image description here

Note that the rounding to 2 decimal places and 'form {table}' of the original string is omitted. Any idea why this is happening?

Aishani Singh
  • 29
  • 5
  • 1
    Probably because of the comma. Since you are reading CSV spark might be treating the things after comma as separate column. try having quotes like `"select .... *100,2"`. It normally shows an error as well but not sure why you don't have any errors – Equinox Jul 18 '22 at 17:22
  • 1
    force add quotes while writing the csv file. or write the csv file with a different delimiter. – samkart Jul 18 '22 at 17:26

0 Answers0