0

we are running Databricks autoloader streaming job by running 24*7. The approach we are trying to follow is to stop the job on weekends and run the vacuum and optimize commands. But not sure how we can stop the job based on the foreachbatch completes.

q=(df.writeStream
    .queryName("sample_stream")
     .trigger(processingTime='10 seconds')
      .outputMode("update")
      .foreachBatch(upsertToDelta)
     .option("checkpointLocation", checkPointLocation)
      .start()
      ) 

0 Answers0