Stack Overflow
Stack Overflow

Questions tagged [apache-nifi]

Apache NiFi is an easy to use, powerful, and reliable distributed system to transform and distribute data.

About

Apache NiFi, formerly Niagara Files, is a powerful data routing and transformation system originally created by the United States National Security Agency (NSA). Flows of data and transformations are expressed using directed graphs of data routing, transformation, and system mediation logic nodes.

NiFi has been designed as a distributed system and, with careful design, works well where limited network connectivity is available - eg a satellite node may receive vast amounts of data, and then samples / summaries can be sent back to a central location over a link with limited bandwidth.

Further, the system maintains extensive data provenance records, and it is possible to inspect flows and see individual data items and the state of them after any transformations at any point in the flow.

Clarification of ownership & license

  1. The Apache Software Foundation is a non-profit organization which owns the intellectual property of Apache NiFi (among many other projects). The Apache License, version 2.0 (additional FAQ) dictates the use and reproduction of the NiFi source code. You can read more about The Apache Way to Sustainable Open Source Success.
  2. You do not need to purchase a license to use, consume, or repurpose the Apache NiFi source code or compiled executables. You only need to adhere to the requirements outlined in the license linked above.
  3. The open source NiFi community (PMC members, committers, contributors, etc.) are available to offer help for free. This is done via:
  1. Multiple vendors do support customers with regards to Apache NiFi. Some provide consulting advice, like answering questions around the software with a certain level of responsiveness or access to experts. Others provide services like flow design & optimization, integration with other software, etc. Some offer their own version of the software with additional features or for specific deployment platforms.
  2. Those vendors may charge for access to their support, warranty, education, services, custom software, etc., but they are not selling you a license to use "Apache NiFi" (if they are charging you just to gain access to Apache NiFi, remember that Apache software is always free to use).

Links

5112 questions
1
vote
1 answer

Nifi Getting specific column value from getHbase result

I am trying to fetch data from Hbase using GetHBase process and the result is as below { "row" : "r5", "cells" : { "cf1:ABC:" : "V1", "cf1:DEF" : "V2", "cf1:HIJ" : "V3", "cf1:KLM" : "V4" } } Now, I want to filter the json for…
Prabhanj
  • 262
  • 2
  • 3
  • 16
1
vote
2 answers

How to access fields inside MongoDB document in Apache NiFi

In my NiFi workflow,I configured a getMongo Processor along with query property . my query is working fine and I am getting back the response from MongoDB in Ni-Fi as shown below { "_id" : { "$oid" : "5dcadc91f027705b1834d6c5" }, "empId" :…
praveen sangalad
  • 338
  • 2
  • 14
1
vote
0 answers

Best way to load large csv ( 200+ GB ) data to Teradata from HDFS?

Question is pretty simple, looking for a way to safely and optimally load very large csv data ( > 200gb ) to Teradata DB. Due to storage restrictions, we have kept the data file in hdfs and need it loaded into a Teradata table. Chopping or splitting…
StrangerThinks
  • 246
  • 4
  • 14
1
vote
1 answer

Why is ExecuteSQLRecord taking a long time to start outputting flow files on large tables?

I am using the ExecuteSQLRecord processor to dump the contents of a large table (100 GB) with 100+ million records. I have set up the properties like below. However, what I am noticing is that it takes a good 45 minutes before I see any flow files…
Vijay Kumar
  • 2,439
  • 2
  • 32
  • 51
1
vote
2 answers

Error trying to parse odata4 from API REST using NIFI

I'm using a Microsoft REST API to query a Azure application, oauth and request goes without problem. The response from InvokeHTTP has this format {"@odata.context":"https://****.dynamics.com/api/data/v9.1/$metadata#endpoint","value":[ here comes the…
Alejandro
  • 519
  • 1
  • 6
  • 32
1
vote
1 answer

NIFI Insert CSV File into Postgres Database with date fields

I would like to insert csv file into my postgres database. I use processors : Getfiles -> Split (cause files are big) -> UpdateAttribute (to add avro.schema) -> ConvertCSvToAvro -> Putdatabaserecord. If i use only string/text fields (in my avro…
Gdseb
  • 11
  • 3
1
vote
1 answer

Nifi Updaterecord removes leading zero from a record

When I use UpdateRecord processor for removing some certain integers from a number and add leading 0. Queued values verify leading 0 is there ,But after the process, I use another Update record processor for adding a new column using previously…
Sachith Muhandiram
  • 2,819
  • 10
  • 45
  • 94
1
vote
1 answer

Apache NiFi: Mapping a csv with multiple columns to create new rows

I found a similar question on stack overflow. This approach worked fine with just a couple of columns But I realised this method is not possible for csv's with a large number of Columns. I have a csv with 75 columns. I decided to follow this…
Himsara Gallege
  • 934
  • 1
  • 8
  • 24
1
vote
1 answer

How to remove specific text from a value using apache nifi

I have a csv file and I need to remove somespecific text from a colum and replace another value for that. My csv date,name 20191106,sachith To do it, I have used UpdateRecord processor with following configurations. /date ->…
Sachith Muhandiram
  • 2,819
  • 10
  • 45
  • 94
1
vote
1 answer

Remove " " from a csv using apache nifi

I have a csv looks like this : id,name,age "1","sachith","29" "2","nalaka","29" I want this to be like : id,name,age 1,sachith,29 2,nalaka,29 For this I have used : ReplaceText Processor. In it I have following settings: Search value :…
Sachith Muhandiram
  • 2,819
  • 10
  • 45
  • 94
1
vote
1 answer

how to set content type csv in apache nifi writer?

i write a nifi custom processor that handle xlsx to csv , and in my code i generate a string csv like this from xlsx, but i don´t know how to send it as csv, and the other processors like inferavroschema from csv can recognize the inpustream as…
javier_orta
  • 457
  • 4
  • 15
1
vote
1 answer

S3 to redshift nifi

I read a while about how to upload my S3 data to Redshift, COPY command, Glue, etc. My pipeline is almost entirely in NIFI something like: extract_data->insert to S3->excecute Lamda process to transform the data or enrich it using Athena, in 2 or 3…
Alejandro
  • 519
  • 1
  • 6
  • 32
1
vote
1 answer

Make group files in S3 instead one from each json - NIFI

I am receiving the result of an API call, make some transformations, and store it in S3, now it stores 1 file for each api call. Resulting in a LOT of files, the flow is: invokeHTTP->Split.json->JoltTransformJSON (I don't need all the…
Alejandro
  • 519
  • 1
  • 6
  • 32
1
vote
1 answer

How to connect to a remote postgresql using apache nifi

I am trying to run apache-nifi one server and postgresql in a remote server. When I try to insert records to remote database, I get following error. java.sql.SQLException: Cannot create JDBC driver of class 'org.postgresql.Driver' for connect…
Sachith Muhandiram
  • 2,819
  • 10
  • 45
  • 94
1
vote
1 answer

Apache Nifi: Counting numbers in a string inside a csv file

I have a csv which contains two columns. But the columns download and upload have a string that can have multiple number of values separated by commas. download,upload "12,3,43","2,33,7,10" "2,1","4,5,6,23,1" I need to add each value inside the…
Himsara Gallege
  • 934
  • 1
  • 8
  • 24
1 2 3
…
99
100