Remove Duplicate In Array Using Pure Javascript Or Typescript

Question

I seem to only find ways to remove duplicates from an array using other JS libraries but I'm looking to do it in pure JS or typescript since I'm working on an Angular project.

My problem is that I may get an array that has duplicate entries, such as this one:

data [0: {Id: 1, Definition: "House"},
      1: {Id: 1, Definition: "House"}]

And I want to filter it out so that I only get

data [0: {Id: 1, Definition: "House"}]

I've tried it using this method but I still get duplicate entries

let uniqueArray = data.filter(function(item, pos) {
    return data.indexOf(item) == pos;
})

This `data.indexOf(item) == pos` will always be true in your case, since find will test Object identity. You need something like: https://jsfiddle.net/khrismuc/hnLowjy2/ — , Sep 26 '18 at 09:17
how do you want to filter out the duplicate entries? Is it only based on the id key? — Ashish, Sep 26 '18 at 09:23
@Ashish either compare the ID or the String in "Definition"... both will work — bobdolan, Sep 26 '18 at 09:26

Ashish · Accepted Answer · 2019-04-15T08:10:34.577

You can achieve what you want in this way:

You can check if the value is already there in your final array using 'some'

data = [{Id: 1, Definition: "House"}, {Id: 1, Definition: "House"}]

const finalOut = []
data.forEach((value) => {
    if (!finalOut.some(x=> (x.Id === value.Id || x.Definition === value.Definition))) 
   {
        finalOut.push(value)
    }
})

You can also achieve this by 'reduce' in clean and elegant way:

const finalOut2 = data.reduce((acc, cur) => acc.some(x=> (x.Id === cur.Id || x.Definition === cur.Definition)) ? acc : acc.concat(cur), [])

As suggested by @Ezequiel using some inside forEach or reduce making the time complexity of order of n square. For smaller sets of data using reduce and some is an elegant approach. But if you are dealing with arrays of very large length, you must avoid order of n square time complexity Here is one such approach with filter:

//Here storing every value of data is inside lookupObj after filtering it. 
//And checking if value is filtered based on if key of the value inside lookupObj

const lookupObj = {} 
const finalOut3 = data.filter(
    x => {
        const is_unique = !(lookupObj[`Id_${x.Id}`] || lookupObj[`Id_${x.Definition}`])
        lookupObj[`Id_${x.Id}`] = true
        lookupObj[`Id_${x.Definition}`] = true
        return is_unique
    }
)

Do be aware that this method runs in O(n²) or quadratic time, because hidden in here is a nested loop. This isn't a problem with small arrays, but gets slow if data.length becomes large. The other methods outlined in this answer run in O(n) or linear time and don't suffer the same fate. — Ezequiel Muns, Sep 26 '18 at 09:51
I have updated the answer with new function to reduce O(n²) complexity. — Ashish, Jun 18 '19 at 09:41

score 2 · Answer 2 · answered Sep 26 '18 at 09:27

2

You can use Id key of the object to get the unique object using array#reduce in an object accumulator and get all the object of this object using Object.values().

let data = [{Id: 1, Definition: "House"},{Id: 1, Definition: "House"}, {Id: 2, Definition: "House2"}, {Id: 2, Definition: "House2"}],
    result = Object.values(data.reduce((r, o) =>  {
      r[o.Id] = r[o.Id] || {...o};
      return r;
    },{}));
console.log(result);

answered Sep 26 '18 at 09:27

Hassan Imam

21,956
5
41
51

sadly I get a syntax error [ts] Property "values" is not available for type "ObjectConstructor" – bobdolan Sep 26 '18 at 09:30
You can use `Object.prototype.values = function(obj) { var res = []; for (var i in obj) { if (obj.hasOwnProperty(i)) { res.push(obj[i]); } } return res; };` – Hassan Imam Sep 26 '18 at 09:33

Ezequiel Muns · Answer 3 · 2018-09-26T09:46:31.463

If the ID is what determines identity, then you could use a Map or a plain Object to dedupe. Same IDs will take the same spot on the map, so you end up with unique objects (if the above assumptions about IDs hold).

let data = [{Id: 1, Definition: "House"}, {Id: 1, Definition: "House"}];
let idToObj = {};
data.forEach((o) => idToObj[o.Id] = o);
let uniqueArray = Object.values(idToObj);

EDIT: In case that you want the same but you may have objects with the same ID but differing in other fields, a Map comes in handy because it can take whole objects as keys:

let data = [{Id: 1, Definition: "House1"}, {Id: 1, Definition: "House2"}];
let map = new Map();
data.forEach((o) => map.set(o, true));
let uniqueArray = [...map.keys()];

Remove Duplicate In Array Using Pure Javascript Or Typescript

3 Answers3

Linked