3

I have created 3 process groups. I want second process group to be executed after first process group is completed, and third process group to be executed after second process group is completed.

Any solution on this please??

priyanka
  • 287
  • 2
  • 5
  • 11

3 Answers3

4

Best practice is to pair "Wait" and "Notify" processors together such that any flows with a strict wait condition are not released until the Notify event is sent by the other flow(s).

In your example, "process group 1" would be responsible for sending the Notify message and "process groups 2 and 3" would each have corresponding Wait processors that essentially loop until the first process group sends notification that it has completed.

aaronsteers
  • 2,277
  • 2
  • 21
  • 38
1

Apache NiFi is designed around a continuous flow assumption. Adapting a continuous flow to job scheduling can be awkward and difficult, because it depends on you providing a definition for when a process group is "completed". NiFi itself has no concept of a process group ever being completed, only stopped or running. There are some things you might be able to do:

  • Merge the output of each process group to a single flowfile that is passed to the next process group (using MergeContent, for example).
  • Schedule a processor at the entry to each group using CRON scheduling, depending on the time to synchronize the flows.
  • Build a custom processor (or ExecuteScript) that polls for the completed conditions of the previous process group.

All of these really boil down to the same issue - how will you know the previous group is "completed"?

James
  • 11,721
  • 2
  • 35
  • 41
0

Each process group has a relationship to connect with other process group.

Try to drag relationship from first process group to second and from second process group to third process group.

It requires you to create output port in first process group to be connected with input port of second processor. If you don't want to use output of first process group in second process group then just write it some directory using 'putFile' processor in second process group and create a independent flow for our second process group.

Try with this and I will let you know if I can find another way to do it.

khushbu kanojia
  • 250
  • 1
  • 3
  • 1
    The use of port is clarily mentioned in apache-nifi user guid as: Port: Dataflows that are constructed using one or more Process Groups need a way to connect a Process Group to other dataflow components. This is achieved by using Ports. A DFM can add any number of Input Ports and Output Ports to a Process Group and name these ports appropriately. https://nifi.apache.org/docs/nifi-docs/html/user-guide.html – khushbu kanojia Jan 31 '17 at 09:23