0

We are using Structured Streaming in Databricks environment, Every time while we run this program - kAFKA - Structured Streaming (DBR6.6, Spark 2.4.5) - Writing to CosmosDB, we are getting the same exception as below just before we do the final joins to save the data to Cosmos DB. We haven't modified any spark specific settings and leveraging the default spark /DBR configurations.

Caused by: org.apache.spark.SparkException:
           Job aborted due to stage failure:
           Task 174 in stage 9353.0 failed 4 times, most recent failure:
           Lost task 174.3 in stage 9353.0 (TID 60863, 10.139.64.9, executor 1): 
           java.lang.IllegalStateException:
           Error reading delta file dbfs:/raw_zone/uffRetail_jointbl_dev_cp1/state/8/174/left-keyToNumValues/1.delta of HDFSStateStoreProvider[id = (op=8,part=174),dir = dbfs:/raw_zone/uffRetail_jointbl_dev_cp1/state/8/174/left-keyToNumValues]: 
           dbfs:/raw_zone/uffRetail_jointbl_dev_cp1/state/8/174/left-keyToNumValues/1.delta does not exist
Caused by: java.io.FileNotFoundException:
           /6455647419774311/raw_zone/uffRetail_jointbl_dev_cp1/state/8/174/left-keyToNumValues/1.delta
James Z
  • 12,209
  • 10
  • 24
  • 44
Vishnu
  • 41
  • 3
  • The file does not exist. Check the path is correct – mck Nov 24 '20 at 09:31
  • It will run for some time, after writing large amounts of data to the delta, usually, we are seeing this issue. Paths are correct since we give it via the program. – Vishnu Nov 24 '20 at 09:46

0 Answers0