Nifi Jolt nested arrays turn into array of custom objects

Question

Am absolutely stumped trying to take nested arrays from JSON input and turn them into an array of objects with proper Keys and values using Nifi Jolt Transform.

The issue I'm reaching is that I need to manually specify the column names. They are not apparent from the JSON response.

Column names are

{
  "column_names": [
    "icao24",
    "callsign",
    "origin_country",
    "time_position",
    "last_contact",
    "long",
    "lat",
    "baro_altitude",
    "on_ground",
    "velocity",
    "true_track",
    "vertical_rate",
    "sensors",
    "geo_altitude",
    "squawk",
    "spi",
    "position_source"
  ]
}

Example input data:

{
  "time": 1675791934,
  "states": [
    [
      "a57b26",
      "N452SM  ",
      "United States",
      1675791621,
      1675791621,
      -105.1168,
      39.9103,
      null,
      true,
      0,
      90,
      null,
      null,
      null,
      null,
      false,
      0
    ],
    [
      "aa56da",
      "UAL1986 ",
      "United States",
      1675791933,
      1675791933,
      -122.1349,
      41.1152,
      10972.8,
      false,
      235.33,
      2.51,
      0,
      null,
      11049,
      null,
      false,
      0
    ]
  ]
}

Expected output:

[
  {
    "icao24": "a57b26",
    "callsign": "N452SM  ",
    "origin_country": "United States",
    "time_position": 1675791621,
    "last_contact": 1675791621,
    "long": -105.1168,
    "lat": 39.9103,
    "baro_altitude": null,
    "on_ground": true,
    "velocity": 0,
    "true_track": 90,
    "vertical_rate": null,
    "sensors": null,
    "geo_altitude": null,
    "squawk": null,
    "spi": false,
    "position_source": 0
  },
  {...}
]

The result that I'm coming up with from shift Jolt spec is...

[
  {
    "operation": "shift",
    "spec": {
      "states": {
        "*": {
          "*": "[&1]"
        }
      }
    }
  }
]

Result:

[
  [
    "a57b26",
    "N452SM  ",
    "United States",
    1675791621,
    1675791621,
    -105.1168,
    39.9103,
    null,
    true,
    0,
    90,
    null,
    null,
    null,
    null,
    false,
    0
  ],
  [
    "aa56da",
    "UAL1986 ",
    "United States",
    1675791933,
    1675791933,
    -122.1349,
    41.1152,
    10972.8,
    false,
    235.33,
    2.51,
    0,
    null,
    11049,
    null,
    false,
    0
  ]
]

I'd be able to possibly figure it out if the column names were sent in the payload, but unfortunately I need to set them manually in the transformation.

I did happen to find a helpful stack of a similar situation, but again, the solution looks like heavy duty regex and I'm just not that quite well versed in Nifi Jolt yet.

Jolt reference first element in array as target name

score 2 · Accepted Answer · edited Feb 08 '23 at 20:31

Ended up solving this on my own :

[
  {
    "operation": "default",
    "spec": {
      "temp": [
        [
          "icao24",
          "callsign",
          "origin_country",
          "time_position",
          "last_contact",
          "long",
          "lat",
          "baro_altitude",
          "on_ground",
          "velocity",
          "true_track",
          "vertical_rate",
          "sensors",
          "geo_altitude",
          "squawk",
          "spi",
          "position_source"
        ]
      ]
    }
  },
  {
    "operation": "shift",
    "spec": {
      "temp": {
        "*": "states[]"
      },
      "states": {
        "*": "states[]"
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "states": {
        "*": {
          "*": "[&1].@(2,[0].[&])"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "0": null,
      "*": "[]"
    }
  }
]

And this is the expected output :

[
  {
    "icao24": "a57b26",
    "callsign": "N452SM  ",
    "origin_country": "United States",
    "time_position": 1675791621,
    "last_contact": 1675791621,
    "long": -105.1168,
    "lat": 39.9103,
    "baro_altitude": null,
    "on_ground": true,
    "velocity": 0,
    "true_track": 90,
    "vertical_rate": null,
    "sensors": null,
    "geo_altitude": null,
    "squawk": null,
    "spi": false,
    "position_source": 0
  },
  {
    "icao24": "aa56da",
    "callsign": "UAL1986 ",
    "origin_country": "United States",
    "time_position": 1675791933,
    "last_contact": 1675791933,
    "long": -122.1349,
    "lat": 41.1152,
    "baro_altitude": 10972.8,
    "on_ground": false,
    "velocity": 235.33,
    "true_track": 2.51,
    "vertical_rate": 0,
    "sensors": null,
    "geo_altitude": 11049,
    "squawk": null,
    "spi": false,
    "position_source": 0
  }
]

@Barbaros could you explain a bit what you did in the third operation to stitch them together. I am getting confused with the levels. [&1] gets the value inside states array. ".@(2,[0].[&]" is getting the values from the first array. So when you go two levels up here is it the new array after the [&1] change or before it. — Scorpy, Feb 10 '23 at 10:06
Your answer could be improved with additional supporting information. Please [edit] to add further details, such as citations or documentation, so that others can confirm that your answer is correct. You can find more information on how to write good answers [in the help center](/help/how-to-answer). — Community, Feb 11 '23 at 23:17

Nifi Jolt nested arrays turn into array of custom objects

1 Answers1