Python, remove duplicates from list of tuples

Question

I have the following list:

[('mail', 167, datetime.datetime(2010, 9, 29)) , 
 ('name', 1317, datetime.datetime(2011, 12, 12)), 
 ('mail', 1045, datetime.datetime(2010, 8, 13)), 
 ('name', 3, datetime.datetime(2011, 11, 3))]

And I want to remove items from the list with coinciding first item in a tuple where date is not the latest. In other words I need to get this:

[('mail', 167, datetime.datetime(2010, 9, 29)) , 
 ('name', 1317, datetime.datetime(2011, 12, 12))]

Björn Pollex · Accepted Answer · 2011-06-29T09:38:15.790

16

You can use a dictionary to store the highest value found for a given key so far:

temp = {}
for key, number, date in input_list:
    if key not in temp: # we see this key for the first time
        temp[key] = (key, number, date)
    else:
        if temp[key][2] < date: # the new date is larger than the old one
            temp[key] = (key, number, date)
result = temp.values()

edited Jun 29 '11 at 09:38

answered Jun 29 '11 at 09:06

Björn Pollex

75,346
28
201
283

Probably not a good idea to use `list` for the var name. Good answer nonetheless. – Shawn Chin Jun 29 '11 at 09:34
1

This is a case where you could consider `collections.namedtuple` to avoid the `[2]`. – Neil G Jun 30 '11 at 02:06

score 2 · Answer 2 · edited May 23 '17 at 12:29

The following approach uses a dictionary to overwrite entries with the same key. Since the list is sorted by the date, older entries get overwritten by newer ones.

temp = {}
for v in sorted(L, key=lambda L: L[2]): # where L is your list
    temp[v[0]] = v
result = temp.values()

Or, for something a lot more compact (but much less readable):

result = dict((v[0],v) for v in sorted(L, key=lambda L: L[2])).values()

Update

This method would be reasonably quick if the list is already (or mostly) sorted by date. If it isn't, and especially if it is a large list, then this may not be the best approach.

For unsorted lists, you will likely get a some performance improvement by sorting by the key first, then the date. i.e. sorted(L, key=lambda L: (L[0],L[2])).

Or, better yet, go for Space_C0wb0y's answer.

+1 This approach is of course superior to mine, if the list is sorted anyway. — Björn Pollex, Jun 29 '11 at 09:42

score 0 · Answer 3 · answered Jun 29 '11 at 09:08

0

d = {}

for item in list:
    if (item[0], item[1]) not in d:
        d[(item[0], item[1])] = item[2]
    else:
        if item[2] > d[(item[0], item[1])]:
            d[(item[0], item[1])] = item[2]

item = [(x[0], x[1], d[x] for x in d.keys()]

answered Jun 29 '11 at 09:08

Patrick

1,138
7
13

utdemir · Answer 4 · 2011-06-29T09:36:52.277

You can do it via sorting the list and getting the highest values by d[2]:

    In [26]: d
    Out[26]: 
    [('mail', 167, datetime.datetime(2010, 9, 29, 0, 0)),
     ('name', 1317, datetime.datetime(2011, 12, 12, 0, 0)),
     ('mail', 1045, datetime.datetime(2010, 8, 13, 0, 0)),
     ('name', 3, datetime.datetime(2011, 11, 3, 0, 0))]

    In [27]: d.sort(key = lambda i: i[2], reverse=True)

    In [28]: d
    Out[28]: 
    [('name', 1317, datetime.datetime(2011, 12, 12, 0, 0)),
     ('name', 3, datetime.datetime(2011, 11, 3, 0, 0)),
     ('mail', 167, datetime.datetime(2010, 9, 29, 0, 0)),
     ('mail', 1045, datetime.datetime(2010, 8, 13, 0, 0))]

    In [29]: [i for pos, i in enumerate(d) if i[0] in [j[0] for j in d[pos+1:]]]
    Out[29]: 
    [('name', 1317, datetime.datetime(2011, 12, 12, 0, 0)),
     ('mail', 167, datetime.datetime(2010, 9, 29, 0, 0))]

score -1 · Answer 5 · answered Jun 29 '11 at 09:23

Here you go.

#!/usr/bin/python2

from pprint import pprint
import datetime

ol = [('mail', 167, datetime.datetime(2010, 9, 29)) , 
     ('name', 1317, datetime.datetime(2011, 12, 12)), 
     ('mail', 1045, datetime.datetime(2010, 8, 13)), 
     ('name', 3, datetime.datetime(2011, 11, 3))]

d = {}

for t in sorted(ol, key=lambda t: (t[0], t[2])):
    d[t[0]] = t
out = d.values()

pprint(out)

That sorts the list using the first and third tuple elements as keys, then removes duplicates by using a hash table.

Python, remove duplicates from list of tuples

5 Answers5

Update