Is there a better way to refactor map and filter code

Question

I have an array of data. array contains over 500k items. I want to map items in the array then fileter it. This is what I have done.

I create a function that filters array tpo get unique values

const getUniqueValues = (array: string[]): string[] => {
  return array.filter((item, index, _array) => _array.indexOf(item) === index);
};

then I pass in the mapped data into the function

const uniqueValues = getUniqueValues(
  editedData.map((bodyItem: any) => bodyItem[index])
).filter(Boolean);

This worked well and faster when the array contains less items. Now, it takes sometimes five to ten minutes to perform action, which isn't good for user experience. Now uniqueValue returns aproximately 210,000 items in array

Is there a better way to perform this in less time?

I have tried array.reduce, not sure of my code tho cos it seems not to solve the problem. if someone could check it out, i'll appreciate

          const uniqueValues = editedData.reduce(
            (accumulator, bodyItem) => {
              const item = bodyItem[index];
              if (!accumulator.includes(item)) {
                accumulator.push(item);
              }
              return accumulator;
            },
            []
          );

Also don't just read the first snippet of the first answer in a duplicate. The above flagged dupe has 150 detailed answers with benchmarks for a lot of the different options. — pilchard, Jun 08 '23 at 09:35

score 1 · Answer 1 · answered Jun 08 '23 at 09:29

1

With that high amount of items, you should be using the builtin Set class, which will get rid of duplicates efficiently. Just replace getUniqueValues with the code below

const getUniqueValues = (array: string[]): string[] => {
  return [...new Set(array)];
};

answered Jun 08 '23 at 09:29

Samathingamajig

11,839
3
12
34

3

vote to close as [duplicate](https://stackoverflow.com/a/33121880/13762301) – pilchard Jun 08 '23 at 09:40

score -1 · Answer 2 · answered Jun 08 '23 at 10:03

can you try this :

const uniqueValues = Array.from(new Set(editedData.map((bodyItem: any) => bodyItem[index]))).filter(Boolean);

new Set(editedData.map((bodyItem: any) => bodyItem[index])) creates a Set, automatically removing any duplicate values.
Array.from() then converts this Set back into an array.
.filter(Boolean) filters out any falsy values.

Is there a better way to refactor map and filter code

2 Answers2