2
[
[0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02], 
[0.02, 0.02, 0.015], 
[0.026, 0.026, 0.02, 0.02, 0.02, 0.015], 
[0.021, 0.021, 0.02, 0.017], [0.077, 0.076, 0.074, 0.055, 0.045, 0.021], 
[0.053, 0.052, 0.051, 0.023, 0.022], 
[0.016, 0.016]
]

The above is a output from a list of list, data['stock'].

I am thinking of removing the duplicate content within each sub-list but can't figure out a way to do it. If you take a look at line 3, you will notice that there are three elements (0.02, 0.02 and 0.015). However, the first 2 elements are actually duplicate and so one of the element is redundant.

Is there a way I could do a check in each sub-list to get rid of the duplicate value while preserving the order?

Please advise!

Lev Levitsky
  • 63,701
  • 20
  • 147
  • 175
Ting Ping
  • 1,145
  • 7
  • 18
  • 34

3 Answers3

3

Looks like the sublists are already sorted, so you can apply itertools.groupby:

In [1]: data = [
   ...: [0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02], 
   ...: [0.02, 0.02, 0.015], 
   ...: [0.026, 0.026, 0.02, 0.02, 0.02, 0.015], 
   ...: [0.021, 0.021, 0.02, 0.017], [0.077, 0.076, 0.074, 0.055, 0.045, 0.021], 
   ...: [0.053, 0.052, 0.051, 0.023, 0.022], 
   ...: [0.016, 0.016]
   ...: ]

In [2]: from itertools import groupby

In [3]: [[k for k, g in groupby(subl)] for subl in data]
Out[3]: 
[[0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02],
 [0.02, 0.015],
 [0.026, 0.02, 0.015],
 [0.021, 0.02, 0.017],
 [0.077, 0.076, 0.074, 0.055, 0.045, 0.021],
 [0.053, 0.052, 0.051, 0.023, 0.022],
 [0.016]]
Lev Levitsky
  • 63,701
  • 20
  • 147
  • 175
  • `[k for k, g in groupby(subl)]` dup-removal should be added to `itertools.receipes`. However I won't up-vote this because I object to answers to questions where the author didn't even make an attempt to solve the problem on their own. – martineau Dec 16 '12 at 13:04
  • @martineau: I am really sorry. This is the 3rd day I am using python. I will read more on the subject. I apologize for that. – Ting Ping Dec 16 '12 at 13:06
  • @martineau Well, thanks for not downvoting :) I answered this because the question is otherwise well stated: copyable sample input is provided, and the desired result is well-described. Ting Ping, that's good attitude :) Docs and search engines are your best friends when programming. – Lev Levitsky Dec 16 '12 at 13:09
  • 1
    I didn't down-vote because I thought your answer had redeeming qualitites. ;-) @Ting Ping: Sorry if my comment seemed harsh. Posting even feeble non-working attempts is fine here -- it at least shows some initiative and that the person isn't just being lazy and asking someone else to do some thinking and write some code for them. Unfortunately there's a lot of that here... – martineau Dec 16 '12 at 13:14
  • I was wondering if OP ever mentioned that the duplicates would be consecutive. In case of non consecutive duplicates, this solution won't work – Abhijit Dec 16 '12 at 15:31
  • @Abhijit The sample input is sorted, and I also mentioned that I assumed sorted input, which means consecutive duplicates. So I guess your solution is a little more general than needed here, which of course is never a bad thing. – Lev Levitsky Dec 16 '12 at 15:56
1

A solution using OrderedDict. This will work for both consecutive and non consecutive duplicates while preserving the order 

>>> from collections import OrderedDict
>>> some_list = [
[0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02],
[0.02, 0.02, 0.015],
[0.026, 0.026, 0.02, 0.02, 0.02, 0.015],
[0.021, 0.021, 0.02, 0.017], [0.077, 0.076, 0.074, 0.055, 0.045, 0.021],
[0.053, 0.052, 0.051, 0.023, 0.022],
[0.016, 0.016]
]
>>> [OrderedDict.fromkeys(e).keys() for e in some_list]
[[0.074, 0.073, 0.072, 0.03, 0.029, 0.024, 0.021, 0.02], [0.02, 0.015], [0.026, 0.02, 0.015], [0.021, 0.02, 0.017], [0.077, 0.076, 0.074, 0.055, 0.045, 0.021], [0.053, 0.052, 0.051, 0.023, 0.022], [0.016]]
Abhijit
  • 62,056
  • 18
  • 131
  • 204
0

Simple anwer:

use list(set(your_list)).

Set creates the unique elements and list() will convert it to list.

>>> lis1 = [1,2,3,4,4,3,2]
>>> print list(set(lis1))
[1, 2, 3, 4]
>>>
Garfield
  • 2,487
  • 4
  • 31
  • 54