Stack Overflow
Stack Overflow

Questions tagged [apache-minifi]

MiNiFi is a subproject of Apache NiFi. It is a complementary data collection approach that supplements the core tenets of NiFi in dataflow management, focusing on the collection of data at the source of its creation.

42 questions
4
votes
1 answer

NiFi memory management

I Just want to understand how we should plan for the capacity of a NiFi instance. We have a NiFi instance which is having around 500 flows. So, the total number of processors enabled on NiFi canvas is around 4000. We do run 2-5 flows simultaneously…
Ankit Tripathi
  • 325
  • 2
  • 12
2
votes
1 answer

How to configure Database Driver Location(s) in a Nifi process when Nifi is runing on a Kuberntes pod

I have deployed Nifi on Kubernetes using cetic/helm-nifi helm chart. I have to insert log data to a MySQL database using a PutDatabaseRecord process. To do that, inside of PutDatabaseRecord process, I have to configure Database Connection URL,…
Kavindu Ariyasinghe
  • 41
  • 5
2
votes
1 answer

Is there a way to programmatically receive target uri for NiFi remote process group?

I'm trying to programmatically create a NiFi flow using the NiFi api. I'm creating a remote process group in an inner flow. The RPG request payload requires the target uri of the remote process group. Is there a way to get that uri programmatically…
Adi Fuchs
  • 205
  • 2
  • 9
2
votes
0 answers

Remote Process Group for NiFi behind NGINX proxy Unauthorized/Anonymous

I have Nifi running behind NGINX reverse proxy. Both are docker containers. I'm developing a flow for MINIFI utilizing Remote Process Group. In future it is expected that MINIFI accesses the remote process group via proxy. My remote process group…
Fedor Petrov
  • 990
  • 9
  • 19
2
votes
2 answers

How to convert a string to a JSON array using NiFi

In NiFi I'm processing a flowfile containing the following attribute: Key: 'my_array' Value: '[u'firstElement', u'secondElement']' I'd like to split flowFile on this array to process each element separately (and then merge). I tried to use…
trivelt
  • 1,913
  • 3
  • 22
  • 44
2
votes
0 answers

Wait-Notify for parallel & sequential processing in Nifi

I have a requirement where I need to execute 4 jobs parallel and when same items job is done in all 4 processors parallely then trigger the next processor for this I have used wait-notify Flow is like 4 parallel jobs -> notify (release signal…
happy
  • 2,550
  • 17
  • 64
  • 109
2
votes
2 answers

How to automate DBCPConnectionPool Controller service in Apache NiFi?

I am new to Apache NiFi and my basic task to ingest data from multiple database sources into HDFS using Apache Nifi. I use 'DBCPConnectionPool' Controller service for database connection pooling service, where I need to provide information such as…
Rishab Prasad
  • 771
  • 1
  • 8
  • 21
1
vote
2 answers

How can I backup nifi processes when restarting Kuberntes pod?

I have deployed Nifi on Kuberntes using cetic/helm-nifi helm chart. We are facing a problem, If nifi pod restarts we lost the all processes that we created. Is there any way to keep a backup of the process in nifi canvas.
Harshana Karunarathna
  • 11
  • 1
1
vote
0 answers

Nifi/Minifi: The rate of the dataflow is exceeding the provenance recording rate. Slowing down flow to accommodate

I'm using Nifi 1.7.0 to create a flow and then I convert it to Minifi 0.5.0. When I run the application, initially it runs well, but after awhile I'm getting this warning message: o.a.n.p.PersistentProvenanceRepository The rate of the dataflow…
Ricardo
  • 11
  • 3
1
vote
0 answers

Is there a way to skip the MiNiFi properties validation?

I'm running a MiNiFi flow with MiNiFi version 0.5.0 and passing the data on to a NiFi flow with NiFi version 1.12.1 Because of the versions gap, the xml that NiFi produces includes a few extra properties for the processor ListFile, that the MiNiFi…
Adi Fuchs
  • 205
  • 2
  • 9
1
vote
1 answer

What causes version of NiFi component to update?

While creating a NiFi flow I'm realizing the versions of the components changing. I understand that the version changes each time the component updates - but what is considered an update of a component? For example, what causes an update in a…
Adi Fuchs
  • 205
  • 2
  • 9
1
vote
0 answers

Can I programmatically get the root uri of a NiFi flow?

I'm programmatically creating a NiFi flow using the NiFi api. I created a process group (I'll call it PG) inside the root process group. Inside the PG I want a Remote Process Group pointing to the root. The Remote Process Group request-payload…
Adi Fuchs
  • 205
  • 2
  • 9
1
vote
0 answers

Minifi C2 server error connecting to nifi to get the template

when i run this link :http//localhost:10080/c2/config?class=raspi3 am able to get the config.yml file present in this path:\minifi-c2-0.4.0\files\raspi3 which is default one! but when I try to fetch the new template from the Nifi then am getting the…
Sunilkumar
  • 11
  • 2
1
vote
2 answers

Apache NiFi interprocess communication

To separate the logic, simplify the main flow and reuse processors groups I'm looking for something to implement interprocess communication logic in Apache NiFi. So far the only thing I was able to find is to use InvokeHTTP in the main flow and…
Shamil
  • 910
  • 4
  • 17
1
vote
1 answer

Failing in MiNiFi-Tutorial, toolkit error: Connection with id ... has invalid destination id

I followed this tutorial: Getting started with MiNiFi Therefore, there is a input port in my root process group and a process group for MiNiFi. The MiNiFi Process Group, contains a Tail File Processor, a Remote Process Groupand a Connection between…
seb
  • 4,279
  • 2
  • 25
  • 36
1
2 3