0

So whenever i am trying to read from a source with stream i get this error "A file referenced in the transaction log cannot be found" and it points to a file that does not exist.

I have tried:

  • Changing the checkpoint location
  • Changing the start location
  • Running "spark._jvm.com.databricks.sql.transaction.tahoe.DeltaLog.clearCache()"

Is there anything else i could do?

Thanks in advance guys n girls!

Lasse
  • 17
  • 9

1 Answers1

0

So! I had another stream that was running and it had the same parent directory as this stream.. this seems to have been a issue.

First stream was looking in: .start("/mnt/dev_stream/first_stream") Second stream was looking in: .start("/mnt/dev_stream/second_stream")

Editing the second stream to look in .start("/mnt/new_dev_stream/new_second_stream") fixed this issue!

Lasse
  • 17
  • 9