Jolt generates null values as transformation output and can't be removed

Question

I have this JSON as input:

{
  "BusinessInfo": {
    "MemberInformation": {
      "Item": [
        {
          "Relation": {
            "Item": [
              {
                "Partner1": "0072938063",
                "Partner2": "0072938064",
                "CategoryId": "BUR001"
              },
              {
                "Partner1": "0072938063",
                "Partner2": "0072937669",
                "CategoryId": "ZCRM06"
              },
              {
                "Partner1": "0072938063",
                "Partner2": "3000011685",
                "CategoryId": "ZCRM06"
              },
              {
                "Partner1": "1020002423",
                "Partner2": "0072938063",
                "CategoryId": "ZCRM01"
              },
              {
                "Partner1": "0072938067",
                "Partner2": "0072938063",
                "CategoryId": "ZCRM04"
              },
              {
                "Partner1": "0072938063",
                "Partner2": "0072937669",
                "CategoryId": "ZCRM04"
              },
              {
                "Partner1": "0072938063",
                "Partner2": "0072938065",
                "CategoryId": "ZCRM04"
              }
            ]
          }
        }
      ]
    }
  }
}

And this Jolt spec:

[
  {
    "operation": "shift",
    "spec": {
      "BusinessInfo": {
        "MemberInformation": {
          "Item": {
            "*": {
              "Relation": {
                "Item": {
                  "*": {
                    "CategoryId": {
                      "ZCRM04": {
                        "@(2,Partner1)": "[&3].Partner1",
                        "@(2,Partner2)": "[&3].Partner2"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  }
]

Having as result, this:

[ null, null, null, null, {
  "Partner1" : "0072938067",
  "Partner2" : "0072938063"
}, {
  "Partner1" : "0072938063",
  "Partner2" : "0072937669"
}, {
  "Partner1" : "0072938063",
  "Partner2" : "0072938065"
} ]

The problem is with the null values generated. I need the same result but without them. I tried remove them adding this operations to the spec:

  {
    "operation": "default",
    "spec": {
      "*": "TRASH"
    }
  },
  {
    "operation": "remove",
    "spec": {
      "TRASH": ""
    }
  }

But doesn't work, the result is almost the same, only now instead of null appears "TRASH":

[ "TRASH", "TRASH", "TRASH", "TRASH", {
  "Partner1" : "0072938067",
  "Partner2" : "0072938063"
}, {
  "Partner1" : "0072938063",
  "Partner2" : "0072937669"
}, {
  "Partner1" : "0072938063",
  "Partner2" : "0072938065"
} ]

What could be wrong? In the first transformation or in the next two operations added. It is possible to avoid this from the first transformation?

score 1 · Accepted Answer · answered May 04 '18 at 10:49

The reason you have nulls in your output array, is because your input "Item" array has 7 elements in it. And you are using those 7 array indicies to determine your output array.

In particular these lines

"Item": {
  "*": {
    "CategoryId": {
      "ZCRM04": {
        "@(2,Partner1)": "[&3].Partner1",
        "@(2,Partner2)": "[&3].Partner2"

Your in your sample input data, "ZCRM04" is the CategoryId in Item array elements 5, 6, 7.

So the first 0 thru 4 elements don't match, but element 5 does, and the spec says copy data from "@(2,Partner1)" to the 5th index of the output array.

"[&3].Partner1"  -->  "[5].Partner1"

So in this case shift will make an output array with 6 elements, where elements 0 thru 4 are null.

With Jolt, if you want to "filter" array elements AND modify/change the contents of those array elements, you need to do two steps.

Spec

[
  {
    "operation": "shift",
    "spec": {
      "BusinessInfo": {
        "MemberInformation": {
          "Item": {
            "*": {
              "Relation": {
                "Item": {
                  "*": {
                    "CategoryId": {
                      "ZCRM04": {
                        // in this first shift
                        //  just "identify" all the 
                        //  Item entries needs
                        "@2": "temp[]"
                      }
                    }
                  }
                }
              }
            }
          }
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "temp": {
        "*": {
          // now for each original "Item" 
          //  "fix" it.
          // In this case only pass thru 
          //  Partner1 and 2
          "Partner1": "[&1].Partner1",
          "Partner2": "[&1].Partner2"
        }
      }
    }
  }
]

Jolt generates null values as transformation output and can't be removed

1 Answers1