How to remove duplicate elements in a list while preserving order?

Question

I just saw a short video from Seth Ladd on Collections.

A Set has only unique elements (are not ordered), but sometimes i need an ordered list and i want to remove all duplicates (the 2nd occurrence of an element e.g. String should be removed from the list)

original input to a list: A, B, C, B, D, A should result in A, B, C, D. I need to keep the order. A result like B, A, D, C would not help me.

Why is there no such function? Because the Dart team made the same decision to not provide one, like the designers of many other standard libraries before them. Do you want such a function? Then you'll either have to implement it yourself, or find a library to do it for you. What is the goal of this question? To find out _why_ this isn't part of the standard library, or to find out _how_ to do it regardless of that fact? — Matt Ball, Feb 02 '13 at 20:05
All those operations on a list would be `O(n^2)` naively. To get better bounds, either sorting or a Set would be used internally. This should be easy to create a helper function for this (if one doesn't exist) using Sets and a probe. (As for why these aren't defined on a list itself - not constructive.) — , Feb 02 '13 at 20:05
Possible duplicate of [How to delete duplicates in a dart List? list.distinct()?](http://stackoverflow.com/questions/12030613/how-to-delete-duplicates-in-a-dart-list-list-distinct?rq=1) — Matt Ball, Feb 02 '13 at 20:07
(Note how removing the irrelevant question and altering the title made this question "Constructive". It's also a duplicate, but perhaps now there will be less down-votes.) — , Feb 02 '13 at 20:12
I would like to know what the reason for the lack of that function is. — Gero, Feb 02 '13 at 20:27
@Gero The reason is it does not exist - this sort of question is generally "Not Constructive". It would require getting a rational from the language/library author(s) or WG. In a few rare occasions this is available - i.e. a proposal was rejected - but it is usually very hard to track down. — , Feb 02 '13 at 22:07
@Gero However, on practicality notes, arguments can be made - in a retrospective view - for separations of concerns and a high (which is bad) cost/benefit ratio. For instance, it's easy to say that all items in a Set must support ordering (and possibly hashing); would it be proper to extend this requirement to items in a List? — , Feb 02 '13 at 22:13
For the record, here is the bug that requests this feature: https://code.google.com/p/dart/issues/detail?id=8274 — Seth Ladd, Feb 03 '13 at 10:50

score 11 · Answer 1 · answered Jul 20 '18 at 22:54

11

Use toSet and then toList

  var ids2 = ["A", "B", "C", "B", "D", "A"];
  var result = ids2.toSet().toList();

[A, B, C, D]

answered Jul 20 '18 at 22:54

atreeon

21,799
13
85
104

1

and what if my list has instances of a class? like `[Instance of Foo], [Instance of Foo], [Instance of Foo]`? Is it possible to remove the duplicates while preserving the object instances? – darksoulsong Jan 19 '21 at 12:41

score 3 · Answer 2 · edited Feb 03 '13 at 13:11

3

Justin Fagnani already gave a good answer. Here is another one:

Iterable distinct(Iterable i) {
  var map = new LinkedHashMap();
  i.forEach((x) { map[x] = true; });
  return map.keys;  // map.keys.toList() would free the map for GC.
}

edited Feb 03 '13 at 13:11

Alexandre Ardhuin

71,959
15
151
132

answered Feb 03 '13 at 01:12

Florian Loitsch

7,698
25
30

1

note: LinkedHashMap is now in dart:collection, and we are probably going to add an InsertionOrderedSet soon. – Florian Loitsch Feb 03 '13 at 01:34

Justin Fagnani · Accepted Answer · 2013-02-04T17:48:42.840

It's fairly easy to implement on your own:

Iterable distinct(Iterable i) {
  var set = new Set();
  return i.where((e) {
    var isNew = !set.contains(e);
    set.add(e);
    return isNew;
  });

It'd be even nicer if Set.add() returned a bool that indicated whether the set was modified:

Iterable distinct(Iterable i) {
  var set = new Set();
  return i.where((e) => set.add(e));
}

You can file feature request bugs of course.

Edit: As Florian points out, the above solution only works if the returned Iterable is only used once. Subsequent uses will return Iterators with no elements, because even element has been seen already on the first use.

To solve this we need to keep a visited set for every Iterator created from the returned Iterable, not just one for the Iterable. We can do that by creating Iterable and Iterator subclasses like with WhereIterable/WhereIterator:

Iterable distinct(Iterable i) => new DistinctIterable(i);

class DistinctIterable<E> extends Iterable<E> {
  final Iterable<E> _iterable;

  DistinctIterable(this._iterable);

  Iterator<E> get iterator {
    return new DistinctIterator<E>(_iterable.iterator);
  }
}

class DistinctIterator<E> extends Iterator<E> {
  final Iterator<E> _iterator;
  final Set<E> _visited = new Set<E>();

  DistinctIterator(this._iterator);

  bool moveNext() {
    while (_iterator.moveNext()) {
      if (!_visited.contains(_iterator.current)) {
        _visited.add(_iterator.current);
        return true;
      }
    }
    return false;
  }

  E get current => _iterator.current;
}

Yes, this is much longer, but it'll work correctly with many-use finite Iterables and one-use infinite Iterables. The infinite iterable use case could easily have problems with memory, which is an argument for not including it in the core lib and forcing developers to make some decisions about what exactly they need.

thx for the answer. I think i needed distinct() like 1 year ago and i wrote my own removeDuplicates(). I thought they need some time to implement the "official" dart-distinct(). And now I hear it is still not available. I just wonder: why? — Gero, Feb 02 '13 at 21:36
There are a lot of things that aren't there yet. Did you file a feature request? — Justin Fagnani, Feb 02 '13 at 23:08
The solution works, but only, if the iterator is used only once. — Florian Loitsch, Feb 03 '13 at 01:12
Accidentally pressed and then spent too much time before updating the comment... The reason it only works once, is, that the returned iterable will redo the 'where' every time it is used. But the 'set' will not be reset. One easy solution is to force the evaluation of the filter with ".toList()". — Florian Loitsch, Feb 03 '13 at 01:21
Tricky, @FlorianLoitsch, thanks for pointing out the problem. I was thinking that this would preserve the laziness but I see the issue. This is why these utilities should be in the core lib :) I'll try to add a correct version. — Justin Fagnani, Feb 03 '13 at 02:46

score 0 · Answer 4 · answered Dec 30 '20 at 06:03

Using generics and generators, you can make a function that works with Iterables of any type

Iterable<T> distinct<T>(Iterable<T> elements) sync* {
  final visited = <T>{};
  for (final el in elements) {
    if (visited.contains(el)) continue;
    yield el;
    visited.add(el);
  }
}

Usage of distinct(["A", "B", "C", "B", "D", "A"])

Or if you'd like to wrap it into an extension:

extension IterableDistinctExt<T> on Iterable<T> {
  Iterable<T> distinct() sync* {
    final visited = <T>{};
    for (final el in this) {
      if (visited.contains(el)) continue;
      yield el;
      visited.add(el);
    }
  }
}

Usage of ["A", "B", "C", "B", "D", "A"].distinct()

How to remove duplicate elements in a list while preserving order?

4 Answers4

Linked