0

Regular (non sync) Jobs are based on commit time. So if I stop any job, what happens to the dependent jobs. In current version, the entire pipeline would stop. Would Sqlake also behave same way or this would happen only for sync jobs?

Ajay C
  • 66
  • 5

1 Answers1

1

With an "unsync'ed" job, if a COPY FROM job was stopped, all downstream transformation jobs would continue to run, but would not process any data. Once the COPY FROM job was restarted, and data ingested, all of the "newly" ingested data would have a $commit_time that would allow the downstream transformation jobs to process that data. In this case, $commit_time would be written as of the time the COPY FROM job was restarted, so the transformation jobs would include this data. This would also apply if you stopped intermediate transformation jobs.

jhall-upsolver
  • 11
  • 3
  • Thanks Jason, so its safe to say only sync jobs will stop. I suppose sync jobs come in effective only for transformation, copy from synced vs un-synced won't matter much as they would continue to ingest data from source as data arrives. – Ajay C Nov 15 '22 at 17:07