0

I am looking for a way to create a critical section in Apache NiFi. What I mean by that is to create a group of processors in which a single FlowFile would be processed exclusively - the next FlowFile would be picked up for processing from a queue when the previous FlowFile finishes with the last processor of the group. Please refer to the picture attached below. process design

Bottom line: At most one FlowFile should be processed within the critical section. Concurrent task parameters applies only to a single processor, not a group of processors.

I want to implement access token management to NiFi API. I would like to keep the token in a cache and also to limit the number of requests to NiFi API.

benji__
  • 47
  • 3
  • 9

1 Answers1

1

You can readily do this in NiFi by putting the processors in the "critical section" into a process group (PG) with an input and output port. The PG can then be configured with a flowfile concurrency of "Single flowfile per node" to make it process only one flowfile at a time. If it needs to be single flowfile per cluster, you can use a loadbalancing strategy of "single node" in a connection before entering the PG.

I'm assuming you are making a custom access token flow because the one already present in the NiFi API doesn't do what you want it to, but if not, do check the documents. It leverages the configured identity providers and gives you an access token that lasts for the configured duration (12 hours by default I think).

enter image description here

Cyrus
  • 2,135
  • 2
  • 11
  • 14
  • Hey, thanks for your response. Where can I set PG concurrency to 1? I thought I can only do that at the flowfile level, which basically doesn't cover my scenario. – benji__ Jan 31 '22 at 08:18
  • @benji__ I see my answer was a bit unclear, so I've fixed it and added a picture. If you hover over the help icon it explains that on the single flowfile setting, no extra flowfiles will be let into the process group until the previous one has finished processing. I believe that is your scenario? – Cyrus Jan 31 '22 at 08:37
  • I see what the issue is. I am using NiFi 1.11.3 and the Process Group Flowfile Concurrency feature was introduced as part of NiFi 1.12.0 release. Thank you for your answer, now I have a reason to upgrade the software. – benji__ Jan 31 '22 at 08:48