0

I am using the File Source stream component to read files from a directory and send a File instance to a custom processor that reads the file and launches a specific task using a TaskLauncher sink. If I drop 5 files in the directory, 5 tasks launch at the same time. What I am trying to achieve is to have each Task executed one after the other, so I need to monitor the state of the Tasks to ensure the prior task has completed before launching another task. What are my options for implementing this? As a side note, I am running this on a Yarn cluster.

Thanks,

-Frank

franktylerva
  • 91
  • 1
  • 4

1 Answers1

0

I think asynchronous task launching by the YARN TaskLauncher could be the reason to make it look like all the tasks are launched at the same time. One possible approach you can try is to have a custom task launcher sink that launches the task and waits for the task status to be completed before it starts processing the next trigger request.

Ilayaperumal Gopinathan
  • 4,099
  • 1
  • 13
  • 12
  • Can you explain, at a high level, the steps in creating a custom task launcher? Again, this would need to interact with the YARN TaskLauncher. – franktylerva Nov 15 '16 at 18:00