remove duplicates Dedup using jolt

Question

i need to remove the duplicates in json message using jolt framework with insuredId and copy the unique insuredId and insuredName to namecode and name respectively,i have acheived removing the duplicates with the insuredId but i dont know how to copy the corresponding Insured name along with it.

Input:

[
  {
    "aircraftId": "ILTA",
    "aircraftTypeCode": "",
    "insuredId": "12020671",
    "insuredName": "Samuel Antony",
    "policyReference": "20081238",
    "uwy": "2017"
  },
  {
    "aircraftId": "ILTA",
    "aircraftTypeCode": "",
    "insuredId": "12020671",
    "insuredName": "Samuel Antony",
    "policyReference": "20081238",
    "uwy": "2017"
  },
  {
    "aircraftId": "ADE",
    "aircraftTypeCode": "",
    "insuredId": "12018832",
    "insuredName": "Mark henry",
    "policyReference": "20082780",
    "uwy": "2017"
  }
]

Jolt Spec :

[
  {
    "operation": "shift",
    "spec": {
      "*": { 
        "insuredId": { 
          "*": "ids.&[]"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "ids": {
        "*": {
          "$": "[#2].nameCode" 
        }
      }
    }
  }
]

Actual output :

[
  {
    "nameCode": "12020671"
  },
  {
    "nameCode": "12018832"
  }
]

Expected Output:

[
  {
    "nameCode": "12020671",
    "name":"Samuel Antony"
  },
  {
    "nameCode": "12018832",
    "name":"Mark henry"
  }
]

Updated Spec (needs to be validated):

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "insuredId": {
          "*": "ids.&[]"
        },
        "insuredName": {
          "*": "insuredNames.&[]"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "ids": {
        "*": {
          "$": "[#2].nameCode"
        }
      },
      "insuredNames": {
        "*": {
          "$": "[#2].name"
        }
      }
    }
  }
]

Dmitry Gorkovets · Answer 1 · 2017-11-07T11:57:02.080

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "insuredId": {
          "*": {
            "@(2,insuredName)": "ids.&[]"
          }
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "ids": {
        "*": {
          "$": "[#2].nameCode",
          "*": "[#2].name"
        }
      }
    }
  },
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "name": "=firstElement(@(1,name))"
      }
    }
  }
]

Explanation.

First operation:

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "insuredId": {
          "*": {
            "@(2,insuredName)": "ids.&[]"
          }
        }
      }
    }
  }
]

It collects map having "insuredId" as key and lists of "insuredName" as values:

{
  "ids" : {
    "12020671" : [ "Samuel Antony", "Samuel Antony" ],
    "12018832" : [ "Mark henry" ]
  }
}

Second operation:

[
  {
    "operation": "shift",
    "spec": {
      "ids": {
        "*": {
          "$": "[#2].nameCode",
          "*": "[#2].name"
        }
      }
    }
  }
]

It converts map to plain list of pairs:

[ {
  "nameCode" : "12018832",
  "name" : "Mark henry"
}, {
  "nameCode" : "12020671",
  "name" : [ "Samuel Antony", "Samuel Antony" ]
} ]

Third operation:

[
  {
    "operation": "modify-overwrite-beta",
    "spec": {
      "*": {
        "name": "=firstElement(@(1,name))"
      }
    }
  }
]

It takes first element of the arrays containing names:

[ {
  "nameCode" : "12018832",
  "name" : "Mark henry"
}, {
  "nameCode" : "12020671",
  "name" : "Samuel Antony"
} ]

Thanks dimitry for yor reply can you please give some comments on understanding the spec.i have also created a updated spec in original post.It gives the expected result but it needs to be validated if it will work for all the cases. — Ravi, Nov 07 '17 at 11:22
Hi Dimitry,Iam getting the error"JOLT Chainr could not find transform class:modify-overwrite-beta at index 2. Iam using Jolt 0.0.20 and it seems Modify wasn't added till Jolt 0.0.22. — Ravi, Nov 07 '17 at 12:16

score 0 · Answer 2 · answered Jul 14 '21 at 11:01

You can use four consecutive shift transformation. Pick individually each desired key within the first step in order to get two lists for each of them. Make the elements unique throughout each list in the second step. In this step, we have two objects with keys. And then convert them to lists again. And group them to the objects without keys with respect to their indexes within each list such as

[
  {
    "operation": "shift",
    "spec": {
      "*": {
        "insuredId": "&",
        "insuredName": "&"
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "*": "&2.&"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": {
          "$": "&2"
        }
      }
    }
  },
  {
    "operation": "shift",
    "spec": {
      "*": {
        "*": "[&].&1"
      }
    }
  }
]

remove duplicates Dedup using jolt

2 Answers2