I am trying to ingest data from my local system to azure data lake storage using NiFi . I have been told to use the putHDFS processor for that , but I do not have Hadoop in my machine . Is there any alternative way to ingest the data or any alternative processor ? I am new to NiFi so any help would be highly appreciated !
Asked
Active
Viewed 257 times
0
-
check this: https://community.hortonworks.com/articles/71916/connecting-to-azure-data-lake-from-a-nifi-dataflow.html – daggett Feb 19 '19 at 08:39
-
Is there any way to do this without using hdf or Hadoop altogether ?? – sayan choudhury Feb 19 '19 at 10:15
-
you can use web api with InvokeHTTP processor: https://learn.microsoft.com/en-us/rest/api/datalakestore/webhdfs-filesystem-apis – daggett Feb 19 '19 at 10:44
-
You don't need hadoop on your machine, PutHDFS is a client talking to a remote hadoop instance. – Bryan Bende Feb 19 '19 at 14:17
-
I am having a hard time figuring out the configuration for putHDFS , the are asking for some xml files and I don't understand which files they are talking about . – sayan choudhury Feb 20 '19 at 10:27
-
Using `PutHDFS` processor is purely a workaround. There is a PR to add native `ADLS` processors to NiFi https://github.com/apache/nifi/pull/2158. Please take a look. – Sivaprasanna Sethuraman Feb 21 '19 at 05:23
-
Thanks for answering , could you please elaborate? It is kind of confusing – sayan choudhury Feb 25 '19 at 07:18