0

Not able to remove white space from SQL query output used in pyspark code.

I tried, trim,ltrim,rtrim,replace (multiple nested also) and regex replace. Any other suggestion.

When trim was used in sql side alone, it worked. But when it is used inside pyspark query no change

pltc
  • 5,836
  • 1
  • 13
  • 31
Reshma Joseph
  • 23
  • 5
  • 3
    Please provide enough code so others can better understand or reproduce the problem. – Community Mar 29 '22 at 17:01
  • @anky tried,not getting trimmed – Reshma Joseph Mar 30 '22 at 00:45
  • compare_query=""" select 'pspk' as data_source, customer_city from old_ticket_df where ticketlineid='f31fd351-3036-4f71-b4e5-cbdf3039429c' """ compare_df=spark.sql(compare_query) compare_df=compare_df.withColumn("customer_city_trim",trim(compare_df['customer_city'])) compare_df.show(truncate=False) – Reshma Joseph Mar 30 '22 at 00:49
  • It looks more like special characters to me than whitespace. – pltc Apr 19 '22 at 17:42

0 Answers0