3

I've been struggling for a few weeks to find a config solution that works how I expect - maybe what I want isn't possible...

Here's what I'm trying to do:

  1. Check an S3 bucket for any files
  2. If there aren't any, don't spin up a cluster just mark the pipeline as "FINISHED" otherwise start the cluster and process the files

What happens is that the pipeline health status ends up being marked as "ERROR" - which IMO it isn't, it's an expected state (there were no files to process), however, the precondition works as expected and the cluster isn't started - which is what I want.

Can anyone post some sample config showing what I need to configure in order to get the pipeline into a "FINISHED" state, rather than "ERROR", when there are no files in the S3 bucket?

Many thanks,

Dave

Dave Chapman
  • 31
  • 2

0 Answers0