Is Apache NiFi slower than StreamSets?
I have created a pipeline which receives data from a Kafka topic and dumps the data in another Kafka topic in both Apache NiFi and StreamSets but StreamSets is way faster than NiFi.
I am using consumekafkaRecord processor in NiFi and KafkaConsumer in StreamSets.
I am very familiar with NiFi. I do not believe NiFi has any advantage over Streamsets for that specific scenario when looked at in terms of per node speed only. NiFi is designed to handle arbitrary sources and sinks which means it generally doesnt and shouldnt assume any transactional behavior of a source. Kafka though does offer a great design pattern around grabbing data, doing things, sending data to kafka or another place and then acking the response. This being an increasingly common and scaleable pattern the NiFi community is launching a NiFi-FN approach which makes both the general data distribution case and a case like this optimal in NiFi. NiFi brings a ton of really important advantages when you look at durability, reliability, diversity of data and sources/sinks, and built-in provenance. If all you need is perf and for this specific case Streamsets is better or for that matter I'd recommend Spark/Spark Streaming. If your needs will expand beyond what is described here and is data distribution/data flow management focused then NiFi will be absolutely the best choice.
