0

I have following scenario with apache-nifi

Flowfile1 :

id,name
1,sachith
2,nalaka
3,muha

Flowfile2:

id,email
1,sachith@email.com
2,nalaka@email.com

After merging these two flowfiles, result should be :

id,name,email
1,sachith,sachith@email.com
2,nalaka,nalaka@email.com
3,muha,

Here most important part is that, 3,muha, row should be there with email field empty.

I have refered Merge two JSON flowfile together in NiFi, which is very similar to my problem. But not with adding empty value for the row.

How to join two CSVs with Apache Nifi

For now I have tried with MergeContent processor, but it appends two flowfiles. Not adding as new columns with unique id.

Very similar question here,Merge two schemas into one in Apache nifi, but it doesn't have much details.

Sachith Muhandiram
  • 2,819
  • 10
  • 45
  • 94

1 Answers1

3

Instead of two flowfiles, if you can use LookupRecord with a CSVRecordLookupService, you should be able to enrich the original flowfile with the details from another.

Doing a streaming enrich is not really a multiple-stream operation per se, as flowfiles might be coming at different times and different places, how would you decide when to send a flowfile on if it hasn't been enriched, or how much memory would it take to store unenriched flow files while they wait for a match?

In the NiFi world, it is assumed there is a "static" place for enrichment data, and using LookupRecord with a LookupService, you can enrich the data as it comes through. Of course some LookupService implementations might support updates (like DatabaseRecordLookupService if additional rows have been added), but the enrichment happens at the point in time when the input flowfile is processed, so LookupRecord only has access to what is available from the LookupService at the time the lookup is made.

mattyb
  • 11,693
  • 15
  • 20