How to iterate json in each Flowfile in Nifi?

Question

For example,there are 8 FFs,and then i’ve convert json to attribute for each FF,as follows:

I've add 5 Properties and Value with EvaluateJsonPath in pic. If i need to convert 1000 multi-attribute,to set 1000 P/V with EvaluateJsonPath is too trouble. What can i do to this easily? Any help is appreciate!TIA

@Lamanus Thank you! Split what?i've split json,as shown above.Can u give me a hint?THANKS — Cong, Sep 10 '19 at 05:46
You can even split the array into each item. Btw, I couldn't understand what you exactly want to do. Can you explain what are you doing and what exactly want to do in more detail, please? — Lamanus, Sep 10 '19 at 06:14
@Lamanus Sorry.To convert JSON to attribute,i can set one Property and Value instead of many P/V in EvaluateJsonPath?I've set 5 P/V in my pic.That is so trouble,if i would set 10000000 P/V in EvaluateJsonPath.Any other suggestion?THANKS — Cong, Sep 10 '19 at 06:43
Your json is not actually json but json array. There are many values with key `id` and you cannot get the attribute from jsonarray as you did. There is no limit for P/V settings but at least it should be from json. I suggest you to split jsonarray into each json, and EvaluateJsonPath for each json, set Property and Values as you wish. You can use SplitText or SplitRecords. — Lamanus, Sep 10 '19 at 07:14
Or try to read the list value by doing `$[0].ID`, i am not sure. you can change the number for list items. See [This](https://nifi.apache.org/docs/nifi-docs/html/expression-language-guide.html#jsonpath) — Lamanus, Sep 10 '19 at 07:15

score 0 · Answer 1 · answered Sep 10 '19 at 13:19

You don't have to (and shouldn't) split the individual JSON objects out of the array if you intend to keep them as a group (i.e. merge them back in). In most cases the split-transform-merge pattern has been replaced by record-based processors such as UpdateRecord or JoltTransformRecord. In your case since the data is JSON you can use JoltTransformJSON with the following spec to change the ID field to ID2 without splitting up the array:

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "ID": "[#2].ID2",
        "*": "[#2].&"
      }
    }
  }
]

Note that you can also do this (especially for non-JSON input/output) with JoltTransformRecord, the only difference being that the spec is applied to each object in the array rather than JoltTransformJSON which applies the spec to the entire array. The JoltTransformRecord spec would look like this:

[
  {
    "operation": "shift",
    "spec": {
      "ID": "ID2",
      "*": "&"
    }
  }
]

Thank you for your suggestion.My plan is that converting JSON to attribute.I've set 5 P/V in my pic.That is so trouble,if i would set 10000000 P/V in EvaluateJsonPath.Can i set one Property and Value instead of all the P/V in EvaluateJsonPath?Any other suggestion?THANKS — Cong, Sep 11 '19 at 00:43
If you want to change all your JSON fields to attributes, you'll likely need a scripted solution using ExecuteScript. However setting that many attributes on a flowfile can easily make NiFi run out of memory. Why do you need them all as attributes? The JOLT processors work on the JSON in the content, not as attributes, which is the preferred way to do transformations. — mattyb, Sep 16 '19 at 13:55

How to iterate json in each Flowfile in Nifi?

1 Answers1