1

I am using a delta table to load data from my dataframe. I am observing that the column values which have a '-' in them, are getting truncated. I tried to check the records in the dataframe that I am loading, by loading them to a csv file and I don't see this issue in the csv file.

Even on doing a DESCRIBE DETAIL DB_NAME.TABLE_NAME, I can see that the createdAt and lastModified columns are having this same issue as shown in the attached screenshot. This seems like some issue with the way table data is being displayed. Can anyone let me know on how to get this fixed? enter image description here

the_data_novice
  • 37
  • 7
  • Strange!. Whats the databricks runtime version? – Jim Todd Apr 03 '21 at 23:07
  • @JimTodd Runtime version 6.4 (Apache Spark 2.4.5, Scala 2.11) – the_data_novice Apr 05 '21 at 08:10
  • This is odd! Are you able to give us some more info on how we can reproduce this? E.g. the csv file? Share more of the code? – Andreas Storvik Strauman Apr 08 '21 at 07:55
  • @AndreasStorvikStrauman This seems to be an issue with the runtime version itself. I could see this issue even if I manually populate the data for these columns, so doesn't look like a code issue. I was able to convince the required stakeholders that there are no impacts to the actual data, and they are good with it. Thanks for checking this though. – the_data_novice Feb 22 '22 at 05:21

0 Answers0