Multiple data collector for a job without duplicating records in streamsets

Question

I have a directory consist of multiple files, and that is shared across multiple data collectors. I have a job to process those files and put it in the destination. Because the records are huge, I want to run the job in multiple data collector. but when I tried I got the duplicate entries in my destination. Is there a way to achieve it without duplicating the records. Thanks

Answer 1

You can use kafka for it. For example:

• Create one pipeline which reads file names and sends them to kafka topic via kafka producer.
• Create pipeline with kafka consumer as an origin and set the consumer group property to it. This pipeline will read filenames and work with files.
• Now you can run multiple pipelines with kafka consumer with the same consumer group. In this case kafka will balance messages within consumer group by itself and you will not be getting duplicates.
• To be sure that you won't have duplicates also set 'acks' = 'all' property to kafka producer.

With this schema you can run as many collectors as your kafka topic partition count. Hope it will help you.

Answer 2

Copying my answer from Ask StreamSets:

At present there is no way to automatically partition directory contents across multiple data collectors.

You could run similar pipelines on multiple data collectors and manually partition the data in the origin using different character ranges in the File Name Pattern configurations. For example, if you had two data collectors, and your file names were distributed across the alphabet, the first instance might process [a-m]* and the second [n-z]*.

One way to do this would be by setting File Name Pattern to a runtime parameter - for example ${FileNamePattern}. You would then set the value for the pattern in the pipeline's parameters tab, or when starting the pipeline via the CLI, API, UI or Control Hub.