Why remove is not working on list of dicts?

Question

I have the following code.

import os

products =  [

        {"Product": "S65-85OS04_M2M_GP211_JC222_R6",
         "PlatformName": "winPc",
         "PlatformVariant": "eeprom",
         "DocGeneration": False,
         "BuildStatus": "Pass",
        },

        {"Product": "SC5-01OS19_GP221_JC302_LTE_MTN",
         "PlatformName": "winPc",
         "PlatformVariant": "flash",
         "DocGeneration": False,
         "BuildStatus": "Fail",
         },

        {"Product": "SC5-01OS01_GP211_JC302_LTE_TMO",
         "PlatformName": "winPc",
         "PlatformVariant": "flash",
         "DocGeneration": False,
         "BuildStatus": "Pass",
         }
    ]

class UTE(object):

    def __init__(self, workspace, products, blackList=None):
        for each in products:
            # print each
            if each['Product'] in blackList:
                products.remove(each)
        for each in products:
            print each["Product"]

if __name__ == '__main__':
    ins = UTE('ws', products, ["SC5-01OS01_GP211_JC302_LTE_TMO", "SC5-01OS19_GP221_JC302_LTE_MTN"])

Now whenever I run this, it removes only one entry in the dict. For example, in this case it is removing the 2nd entry and that is SC5-01OS19_GP221_JC302_LTE_MTN. I believe this is something related to shallow copy..Am I right ?? If not then how solve this issue?

score 1 · Answer 1 · answered Apr 26 '16 at 05:20

1

for each in products:
    # print each
    if each['Product'] in blackList:
        products.remove(each)

Here you are modifying the list while iterating over it. That would give you unexpected result. A quick fix is: iterate over a copy of the list:

for each in products[:]:

answered Apr 26 '16 at 05:20

Yu Hao

119,891
44
235
294

But if I do `a = [1, 2, 3, 4, 5, 6, 7, 8]` and do `for e in a: if e % 2 == 0: a.remove(e)`, then it works fine..Why?? – Mayukh Sarkar Apr 26 '16 at 05:31
@MayukhSarkar *Unexpected* -> sometimes it *seems* to work fine, sometimes it doesn't. – Yu Hao Apr 26 '16 at 05:37

AKS · Accepted Answer · 2016-04-26T05:56:48.903

You are removing the entry from the list while iterating over the same list. You could simply use list comprehension to keep the elements which are required:

products = [product for product in products 
            if product.get('Product', None) not in blacklist]

Or using filter:

products = filter(lambda product: product.get('Product', None) not in blacklist, products)

Update regarding remove from a = [1, 2, 3, 4, 5, 6, 7, 8]:

You say following code works:

a = [1, 2, 3, 4, 5, 6, 7, 8]
for e in a:
    if e % 2 = 0:
        a.remove(e)

print(a) # [1, 3, 5, 7]

Well, it doesn't work but it is just an illusion, let's add some print statements to understand:

for e in a:
    print 'Next Item: ', e
    if e % 2 = 0:
        a.remove(e)

    print 'Current List: ', a

And, here is the output from the print statements:

Next Item:  1
Current List:  [1, 2, 3, 4, 5, 6, 7, 8]
Next Item:  2
Current List:  [1, 3, 4, 5, 6, 7, 8]
Next Item:  4
Current List:  [1, 3, 5, 6, 7, 8]
Next Item:  6
Current List:  [1, 3, 5, 7, 8]
Next Item:  8
Current List:  [1, 3, 5, 7]

As, you can see, just before the 3rd iteration the a is updated and 3 is shifted to the 2nd position and since 2nd position is already iterated 3 never comes in the loop. And, similarly for others so the loop doesn't run 8 times but only 5 times.

If you change the order of values in a you won't get the same behavior:

a = [1, 4, 2, 3, 6, 8, 7, 5]

Now, run the above `for loop again:

Next Item:  1
Current List:  [1, 4, 2, 3, 6, 8, 7, 5]
Next Item:  4
Current List:  [1, 2, 3, 6, 8, 7, 5]
Next Item:  3
Current List:  [1, 2, 3, 6, 8, 7, 5]
Next Item:  6
Current List:  [1, 2, 3, 8, 7, 5]
Next Item:  7
Current List:  [1, 2, 3, 8, 7, 5]
Next Item:  5
Current List:  [1, 2, 3, 8, 7, 5]

So here at the end of the list the value of a is [1, 2, 3, 8, 7, 5].

So like I said earlier: It doesn't work but it is just an illusion that it does

But if I do `a = [1, 2, 3, 4, 5, 6, 7, 8]` and do `for e in a: if e % 2 == 0: a.remove(e)`, then it works fine..Why?? — Mayukh Sarkar, Apr 26 '16 at 05:32
@MayukhSarkar I updated my answer with proper reasoning. Please have a look. — AKS, Apr 26 '16 at 05:57

score 0 · Answer 3 · edited May 23 '17 at 12:31

As explained by AKS, you can use list comprehensions to filter out list elements. In addition, you can use the built in filter function:

some_integers = range(15)

# via filter
odd_integers = filter(lambda i: i%2 != 0, some_integers)

# via list comprehension
odd_integers = [num for num in some_integers if num %2 != 0]

Ultimately, the issue you're running into is that you are modifying your list as you iterate through it. This has been discussed many times, for example: Modifying list while iterating

Why remove is not working on list of dicts?

3 Answers3