How can I pass a dynamic attribute to Nifi EvaluateJsonPath?

Question

Currently, I'm building a 3rd party automated data collection process, through NiFi that runs automatically syncing data to datastax Astra. Here is a sample api return I got from a third party.

{
    "id": "123",
    "project_name": "ALS Data",
    "descriptions": "This is my first project",
    "members": [
        {
            "id": "345",
            "full_name": "John",
            "role": "Front-End"
        },
        {
            "id": "456",
            "full_name": "Iko Juno",
            "role": "Backend"
        }
    ]
}

• I want to filter through the list of each member and call the api of datastax Astra to save the data there. Enpoint API in datastax Astra {BaseURL}/api/rest/v1/keyspaces/{keyspaces_name}/tables/{table_name}/rows.

• In NiFi, I tried using EvaluJsonPath to get each value of the Members array (in the api return I mentioned above). However, I find using EvaluateJsonPath doesn't seem to work as I only get exactly 1 object that I pass in. Example: $.members[0].id

How can i pass a dynamic attribute to EvaluateJsonPath so that I can get all the data in the returned array and import them into datastax Astra through Astra's built-in api. Or any other way to handle this problem. Thanks a lot!

Answer 1

If I understand right you want to extract the members array from the API response and then do further processing.

So to replicate your case I made use of GenerateFlowFile which generates me a flowfile, I copied the API response you shared above and added it to the properties tab.

Next I make use of the SplitJson processor, this takes care of splitting the single flowfile into multiple files based on the split property provided. In this case as I wanted to get all the values from the members array, I used $.members

Now running the process once you can see the below output at each stage

Generate FlowFile

SplitJson

You would have two flowfiles at the end of the SplitJson as the API response had two values in the members array. After this you can do further processing as needed.