I'm downloading a file source and creating and stream to process line by line to finally sink into HDFS. For that purpose I'm using Spring Cloud Dataflow + Kafka.
Question: is there any way to know when the complete file has been sunk into HDFS to trigger an event?
is there any way to know when the complete file has been sunk into HDFS to trigger an event?
This type of use-case typically falls under task/batch as opposed to streaming pipeline. If you build a filehdfs task (batch-job) application, you could then have a stream listening to various task-events in order to make further downstream decisions or data processing.
Please refer to "Subscribing to Task/Batch Events" from the reference guide for more details.
