Efficiently remove duplicates and similar elements from a list

Question

I have a List which contains equal-length-sequences (In this example the length of the sequences is 2, but it can be longer than that in other cases) of numbers, for example:

[[3, 2], [0, 6], [6, 0], [7, 7], [0, 5], [7, 7]]

Now I want to remove all duplicates ([7, 7] appears twice) but also remove all reverse duplicates ([0, 6] and [6, 0]).

My current code is this:

def getUniqueShapes(shapes):
    unique = []
    for shape in shapes:
        if shape not in unique and reversed(shape) not in unique:
            shapes.append(shape)

    print(unique)

My expected output for this example is

[[3, 2], [0, 6], [7, 7], [0, 5]]

Because of the not in, this algorithm runs with very poor speed. Is there an easy way I can apply such a condition to the list with a better time complexity?

@jonrsharpe I'm not sure if sets would work since I don't only want to check if list elements are equal, but also if the reversed element is equal to another one. — Ian Rehwinkel, Sep 12 '19 at 17:33
What do you mean by remove? Remove both of them or just on of them. If one of them which one? — Poojan, Sep 12 '19 at 17:33
@jonrsharpe yes, for this case it would work, but once the length exceeds 2, the order cannot be preserved... (`[1, 2, 3] =! [2, 1, 3]`) — Ian Rehwinkel, Sep 12 '19 at 17:35
@C.Nivs I didn't have an output because the code ran forever because I wasn't appending to `unique` as @GoingTharn pointed out. — Ian Rehwinkel, Sep 12 '19 at 17:38
@C.Nivs For this example `[[3, 2], [0, 6], [7, 7], [0, 5]]` or `[[3, 2], [6, 0], [7, 7], [0, 5]]`. — Ian Rehwinkel, Sep 12 '19 at 17:42

score 4 · Accepted Answer · edited Sep 12 '19 at 17:43

4

Assuming in case of [1, 0], [0, 1] both means same.
You need to convert lists to tuple first because list is not hashable.
Also output is set of tuples you can easily convert it to list of lists.

>>> l = [[1, 2], [2, 1], [3, 4], [5, 6], [5, 6]]
>>> set(map(tuple, map(sorted, l)))
{(1, 2), (3, 4), (5, 6)}

Edit: This will work for any length cases elements.

edited Sep 12 '19 at 17:43

Engineero

12,340
5
53
75

answered Sep 12 '19 at 17:35

Poojan

3,366
2
17
33

Efficiently remove duplicates and similar elements from a list

1 Answers1