Stack Overflow
Stack Overflow

Questions tagged [apache-nifi]

Apache NiFi is an easy to use, powerful, and reliable distributed system to transform and distribute data.

About

Apache NiFi, formerly Niagara Files, is a powerful data routing and transformation system originally created by the United States National Security Agency (NSA). Flows of data and transformations are expressed using directed graphs of data routing, transformation, and system mediation logic nodes.

NiFi has been designed as a distributed system and, with careful design, works well where limited network connectivity is available - eg a satellite node may receive vast amounts of data, and then samples / summaries can be sent back to a central location over a link with limited bandwidth.

Further, the system maintains extensive data provenance records, and it is possible to inspect flows and see individual data items and the state of them after any transformations at any point in the flow.

Clarification of ownership & license

  1. The Apache Software Foundation is a non-profit organization which owns the intellectual property of Apache NiFi (among many other projects). The Apache License, version 2.0 (additional FAQ) dictates the use and reproduction of the NiFi source code. You can read more about The Apache Way to Sustainable Open Source Success.
  2. You do not need to purchase a license to use, consume, or repurpose the Apache NiFi source code or compiled executables. You only need to adhere to the requirements outlined in the license linked above.
  3. The open source NiFi community (PMC members, committers, contributors, etc.) are available to offer help for free. This is done via:
  1. Multiple vendors do support customers with regards to Apache NiFi. Some provide consulting advice, like answering questions around the software with a certain level of responsiveness or access to experts. Others provide services like flow design & optimization, integration with other software, etc. Some offer their own version of the software with additional features or for specific deployment platforms.
  2. Those vendors may charge for access to their support, warranty, education, services, custom software, etc., but they are not selling you a license to use "Apache NiFi" (if they are charging you just to gain access to Apache NiFi, remember that Apache software is always free to use).

Links

5112 questions
1
vote
3 answers

How can I two-phase split large Json File on NiFi

I'm using NiFi for recover and put to Kafka many data. I'm actually in test phase and i'm using a large Json file. My Json file countains 500K recordings. Actually, I have a processor getFile for get the file and a SplitJson. JsonPath Expression :…
BastienB
  • 180
  • 2
  • 14
1
vote
1 answer

Why is Zookeeper not re-electing new leader in Apache Nifi Cluster?

Following is my architecture 2 Servers: Server 1: running Apache Nifi + Zookeeper (Not embedded) Server 2: running Apache Nifi + Zookeeper (Not embedded) To test failovers, I close down the Server that has been selected as Cluster Coordinator In…
Jaskaran Singh Puri
  • 729
  • 2
  • 11
  • 37
1
vote
1 answer

Generate UUID5 in NiFi

In NiFi, I have a flow file with an attribute RSID. I need to generate a UUID v5 based on RSID and add it as an attribute to the flow file. This uuid needs to be based on RSID because some reports will have the same RSID and need to thus have the…
carousallie
  • 776
  • 1
  • 7
  • 25
1
vote
1 answer

nifi Json data using routeonattributeto filter attribute or attribute value

I am currently working to consume data using Nifi to read tealium event stream and load into HDFS. Need help in filtering the data when source misses to send data for attribute. {"account":"newtv","twitter:description":"Discover when your favorite…
Deepak
  • 13
  • 1
  • 6
1
vote
1 answer

Nifi processor runs recursively

I am exploring nifi, as of I have created processor group with number of processor which basically select data from Oracle DB and insert in to mongoDB. the the flow works as expected. The flow is QueryDatabaseTable -> SplitAvro -> ConvertAvorToJson…
Hitesh Ghuge
  • 793
  • 2
  • 10
  • 39
1
vote
2 answers

NiFi - Increasing QueryDatabaseTable performance to outperform Sqoop

I am using QueryDatabaseTable to read from a PDA/Netezza table that has about 70 million rows. Nifi is set up as a single node instance with bootstrap.conf edited to give 16g RAM. QueryDatabase table is using defaults except for Max Rows Per Flow…
bp2010
  • 2,342
  • 17
  • 34
1
vote
1 answer

Getting Nifi to use 'alter table' in Hive

I have Nifi set to pick up files from an SFTP and then drop them into HDFS folders based on the filename. From there, it needs to send the command alter table ${dbname}.${tablename} add partition (year=${year}, date='${date}'); I get that I need to…
lengthy_preamble
  • 404
  • 3
  • 14
  • 35
1
vote
2 answers

How to reorder CSV columns in Apache NiFi

Reorder column in a csv in apache nifi. Input - I have multiple files which have same columns but are in different order. Output - Scrape some columns and store in same order.
Vinayak Mishra
  • 341
  • 4
  • 11
1
vote
0 answers

Double Quotes removed from JSON message when passing an attribute to ExecuteScriptCommand Processor

I am using two steps in Nifi: Step 1. InvokeHTTP processor to make a REST API call and receive a JSON response, "Put Response Body In Attribute" used to store the JSON response into an attribute. Step 2. ExecuteStreamCommand processor to invoke…
Kavipriya M
  • 131
  • 5
1
vote
1 answer

Nifi - Processing Huge json array of records into single records

I am using a invoke HTTP processor that gives an array of JSON objects. the size of the response is between 2-3 GB. The response looks something like this - [ { "id": 17, "name": "ONE by AOL: Video" }, { "id": 63, "name": "Adform" }…
gashu
  • 863
  • 2
  • 10
  • 21
1
vote
3 answers

how to do record count of .csv and attach that count with email processor to send anyone?

What I tried so far is as follow: kindly describe in details how many ways you can read .csv, what I learned so far you need to provide a schema name for the file and then define a schema in the form of .avro or text. is it necessary to provide…
Izhar Ali
  • 17
  • 2
1
vote
1 answer

NiFi - QueryDatabaseTable for 100s of tables - Generic Solution

Currently have a flow using QueryDatabaseTable which reads from a DB and puts the data into HDFS. Decided to use QueryDatabaseTable because: of the state kept for using it for delta loads also the fine tuning when tables are in the 100s of…
bp2010
  • 2,342
  • 17
  • 34
1
vote
1 answer

File in docker volume not updating

Hi I have a Nifi docker container stopped and I want to update a property file. Whenever I update a field, when I run docker-compose start it doesn't update the property file. How can this be possible? here is my docker compose: version:…
3nomis
  • 1,175
  • 1
  • 9
  • 30
1
vote
0 answers

Session.Create(FlowFile) Transfers with no content

I am trying to build a Excecute Script Processor for Nifi. It handles a JSON file, splits it and sends it to the next processor, which is an MongoDB writer. The logic works so far. The main problem is, that I cannot get the processor to create and…
Derek Haynes
  • 125
  • 1
  • 2
  • 13
1
vote
1 answer

unable to merge content in NIFI using merge content processor

I have a text file and i need to merge text files into one single file using NIFI when I am using merge content processor data is appending unable to merge example :- if my text file has data like this in file1:- 2201 and in file2:- 2202 after…
Rahul Varma
  • 550
  • 5
  • 23
1 2 3
…
99
100