2

So, I am very very new to all the Apache Frameworks I am trying to use. I want your suggestions on a couple of workflow design for an IoT streaming application:

  1. As we have NiFi connectors available for Flink, and we can easily use Beam abstraction over Flink. Can I use NiFi as the dataflow tool to drive data from MiNiFi to Flink Cluster (here store it in-memory or something) and then use Beam Pipeline to process the data further.
  2. Is there any NiFi connector for the beam ? If not can we do so? So, we directly stream data from NiFi to the Beam job (running on a Flink Cluster)

I am still in the early design phase, it would be great if we can discuss possible workarounds. Let me know if you need any other details.

user4157124
  • 2,809
  • 13
  • 27
  • 42
Subham Agrawal
  • 55
  • 9
  • Can you describe your use case a bit more? – Turker Tunali Sep 09 '20 at 08:31
  • I want to import data from Apache NiFi to Apache Beam (running on Flink Cluster- using the new Portable runner ) @TurkerTunali – Subham Agrawal Sep 10 '20 at 06:38
  • 1
    It really depends on the specifics like data rate and volume. I have seen the general setups NiFi + Flink and Flink + Beam running fine in production, so the combination of the three might be well possible. Could you also add why you want to have Beam in this architecture? – Arvid Heise Sep 10 '20 at 20:26
  • beam is just for making pipelining easy and "pythonic" (i think Flink Python is still at a quite early stage). Can you please describe more - how can we use the combination (NiFi+Flink+Beam) without the NiFi & Beam connector (if we can compromise a bit on the processing) @ArvidHeise – Subham Agrawal Sep 13 '20 at 03:18

0 Answers0