10

I am developing a solution in Java which communicates with a set of devices through REST APIs which belongs to different vendors. So for each vendor, there are a set of processes that I have to perform inside my solution. However, these processes will be changed according to each vendor. Following are the high-level processes that need to be performed.

  • Retrieve an XML file from a folder
  • Process the XML file
  • Perform some image processing
  • Schedule a job and execute it on the scheduled time
  • Storing data on a MySQL DB and perform some REST calls to outside APIs

So for one vendor might have all of the above processes. But for another, might not have some processes (Eg: Image processing). Following things should be able to obtain from the selected solution.

  • I should be able to create custom workflows for new vendors
  • Need to identify any failures that have been occurred within the workflow and perform retry mechanisms.
  • Should be able to execute some functions parallelly (Eg: Image processing)
  • Scalable
  • Opensource

So I was told to look into workflow managers like Nifi/Airflow/Falcon. I did some research on them but couldn't finalize the most suitable solution.

NOTE: There is NO requirement to use Hadoop or any other cluster and data flow frequency is not that high

Currently, I am thinking of using Nifi. What would be the best solution for my use case?

Selaka Nanayakkara
  • 3,296
  • 1
  • 22
  • 42

2 Answers2

15

Apache NiFi is not a workflow manager in the way the Apache Airflow or Apache Oozie are. It is a data flow tool - it routes and transforms data. It is not intended to schedule jobs but rather allows you to collect data from multiple locations, define discrete steps to process that data and route that data to different destinations.

Apache Falcon is again different in that it allows you to more easily define and manage HDFS datasets. It is effectively data management within a HDFS cluster.

Based on your description, NiFi would be useful addition to your requirements. It would be able to collect your XML file, process in it in some manner, store the data in MySQL, and perform REST calls. It would also be easily configurable for new vendors, and tolerates failures well. It performs most functions in parallel and can be scaled into a clustered NiFi with multiple host machines. It was designed with performance and reliability in mind.

What I am unsure about is the ability to perform image processing. There are some processors (extract image metadata, resize image) but otherwise you would need to develop a new processor in Java - which is relatively easy. Or, if the image processing uses Python or some other scripting language, you can use one of the ExecuteScript processors.

'Scheduling jobs' using NiFi is not recommended.

Full disclosure: I am an Apache NiFi contributor.

returntosender404
  • 498
  • 3
  • 9
  • Can I please know the reason why it is not recommended to schedule jobs using NiFi? – diyath.nelaka Nov 14 '18 at 06:11
  • 1
    I should clarify - I don't believe dataflows within NiFi itself were generally intended to be scheduled. In other words, NiFi was designed for live data feeds. Using an external tool to trigger/schedule dataflows within NiFi via the API (treating NiFi like a batch data processor) was not an intended use case. Using NiFi to schedule external jobs is entirely possible but I doubt it's easy to fully manage external systems - execute, check for success and handle errors using only NiFi processor/s? I'd wager there are better better tools for that. – returntosender404 Nov 15 '18 at 22:17
  • Thanks @returntosender404 for your detailed explanation. I also have same type of requirement and using NIFI. – NS Saravanan Feb 19 '21 at 06:24
5

I am using nifi with an OP's similar use case. Regarding scheduling, I like how nifi works with Kafka, I have some scripts scheduled to run with a crontab frequency, just adding the message into Kafka topics, which topic is listened by nifi, then starts the orchestration for loading, transforming, fetching, indexing, storing, etc, also, you can always handle HttpRequest so you can make kinda "webhook receivers" in order to trigger a process from an external HTTP POST, once again, for simple deployments (these ones you plug and play in a single machine) cronjob nails the task. For image processing, I have an OCR image reader with python connected with an ExecuteScript processor and one facial reckon with opencv with ExecuteCommand processor, the automatic nifi's back-pressure has solved many of the problems I ran by only running the python script and the command by itself.

panchicore
  • 11,451
  • 12
  • 74
  • 100