0

I am writing a SCDF spi implementation for supporting stream and task application. As part of this we need to perform some clean up operations once the task finishes.

Could someone provide info on whether SCDF will be getting a callback on task completion. If not then what are the alternative ways to perform cleanup.

Ankit Bansal
  • 4,962
  • 1
  • 23
  • 40
  • What type of cleanup do you want to do? – Michael Minella Jan 22 '18 at 16:04
  • We are writing a platform for SCDF stream/tasks. As part of this we require to launch some processes from SCDF for each task execution. Ideally those processes should be stopped as soon as task execution is finished but SCDF is not getting any callback for task completion. Since, task can be any user app, these processes must be invoked from SCDF and not from task app. – Ankit Bansal Jan 23 '18 at 06:14
  • That type of cleanup can be done via a `TaskExecutionListener`. The `TaskExecutionListener#onTaskEnd` or `TaskExecutionListner#onTaskFail` could both be used to clean up those processes and is the method I'd recommend. – Michael Minella Jan 23 '18 at 12:55
  • The issue is TaskExecutionListener is part of spring-cloud-task but not spring cloud deployer spi. We need something at spring cloud dataflow/deployer so that we are not dependent on task application since that will be user application.I can see point 2 from Sabby is closest or the other option is SCDF poll db. – Ankit Bansal Jan 23 '18 at 16:32
  • I'd argue that cleaning up the resources consumed by a task application is that application's responsibility... – Michael Minella Jan 23 '18 at 16:34
  • I wouldn't say there is anything wrong with this approach. But, since SCDF is the one that's launching task, there are scenarios where SCDF will need a callback on task completion else task extensibility will become limited. – Ankit Bansal Jan 23 '18 at 16:43
  • Do keep in mind, SCDF doesn't do any of the work. The other proposals are still just using Spring Cloud Stream or Spring Cloud Task components and have nothing to do with SCDF itself. – Michael Minella Jan 23 '18 at 16:44
  • Let us [continue this discussion in chat](http://chat.stackoverflow.com/rooms/163734/discussion-between-ankit-bansal-and-michael-minella). – Ankit Bansal Jan 23 '18 at 16:44

1 Answers1

1

A Task is a short-lived and finite operation. Depending on what you're trying to accomplish, you can do one of the following to invoke any custom cleanup routine.

1) A task running a batch-job and in that job, you can define "n" number of steps as part of a workflow, and upon successful upstream steps, the last step could invoke the cleanup routine.

2) You can have a stream in SCDF listening to Task-complete events (a batch-job example here), which can finally kickoff another task/job to invoke the cleanup routine.

3) You can define a composed-task graph (via Dashboard/shell) where each of the steps (aka tasks) can run its intended operation, and upon successful transition or failure event, you get the opportunity to kick off the cleanup routine.

Sabby Anandan
  • 5,636
  • 2
  • 12
  • 21