1

I was exploring NiFi documentation. I must agree that it is one of the well documented open-source projects out there.

My understanding is that the processor runs on all nodes of the cluster. However, I was wondering about how the content is distributed among cluster nodes when we use content pulling processors like FetchS3Object, FetchHDFS etc. In processor like FetchHDFS or FetchSFTP, will all nodes make connection to the source? Does it split the content and fetch from multiple nodes or One node fetched the content and load balance it in the downstream queues?

pushpavanthar
  • 819
  • 6
  • 20

2 Answers2

1

I think this document has an answer to your question:

https://community.hortonworks.com/articles/16120/how-do-i-distribute-data-across-a-nifi-cluster.html

enter image description here

For other file stores the idea is the same.

will all nodes make connection to the source?

Yes. If you did not limit your processor to work only on primary node - it runs on all nodes.

enter image description here

daggett
  • 26,404
  • 3
  • 40
  • 56
  • Thanks `daggett`. Meanwhile I found a nice article which explained clearly the enhancements made in this area in version 1.8. https://pierrevillard.com/2018/10/29/nifi-1-8-revolutionizing-the-list-fetch-pattern-and-more/ – pushpavanthar Feb 07 '19 at 10:14
1

The answer by @dagget has traditionally been the approach to handle this situation, often referred to as the "list + fetch" pattern. List processor runs on Primary Node only, listings sent to RPG to re-distribute across the cluster, input port receives listings and connect to a fetch processor running on all nodes fetching in parallel.

In 1.8.0 there are now load balanced connections which remove the need for the RPG. You would still run the List processor on Primary Node only, but then connect it directly to the Fetch processors, and configure the queue in between to load balance.

Bryan Bende
  • 18,320
  • 1
  • 28
  • 39