1

In a typical pipeline scenario, say I have a bounded stream, where I read from a file. Is there a way in Jet where I can subscribe to an "OnComplete" event, which will be triggered once the stream is written to sink? I don't seem to find such an option. I want something like below.

p.readFrom(fileSource)
 .writeTo(Sinks.logger())
 .onComplete(doSomething()); 

Edit : 
Reference for the comment.

BatchStage stage = p.readFrom(source).map(transform);
 
stage.map(enrich)
        .writeTo(Sinks.filesBuilder(folder1).build())
        .onComplete(doSomething1());
 
stage.map(enrich2)
.aggregate(...)
.writeTo(Sinks.filesBuilder(folder2).build())
.onComplete(doSomething2());
Rajesh
  • 153
  • 6

1 Answers1

0

After you submit a job:

JetInstance jet = ...
Job job = jet.newJob(p);

You can retrieve a CompletableFuture associated with the job and wait for its completion. Alternatively, you can call

job.join();

which is equivalent to

job.getFuture().join()

It will wait until the job is completed.

František Hartman
  • 14,436
  • 2
  • 40
  • 60
  • Thanks. Yes, this is what available a Job level. I am writing to mutiple sinks. I am looking for an onComplete at Sink level. I have found 'onDestry' at sink level,. Not sure if I can use that. – Rajesh Aug 11 '20 at 05:42
  • 1
    The `onDestroy` on sink level, which is available on `SinkBuilder`, is called once for each parallel worker, to clean up resources of that worker. Other workers can still be writing. Also it's not available for built-in sources, only for your custom sources. – Oliv Aug 11 '20 at 06:32
  • @Rajesh what sink are you using and what would you like to do when it's finished? – Can Gencer Aug 11 '20 at 07:28
  • I want to read data from a file. Then I would do some transformations and write to a file. Meanwhile with the same stage at source, I do some aggregation and write to another file. All this happens as part of 1 single job. I want to trigger some other operation after step 1 is complete. I dont want to wait till entire job is complete. I could do this with multiple jobs, but I want to do this in a single read. I hope I am clear. I'll add some code to the description, if that helps. Thanks – Rajesh Aug 11 '20 at 10:03
  • continued... Worse case I have to wait for the entire job to complete. Or I have to do these as separate jobs I guess. – Rajesh Aug 11 '20 at 10:20