11

I have this generator that yields lists:

def gen():
    state = [None]

    for i in range(5):
        state[0] = i
        yield state

And here's the output, when I call it:

>>> list(gen())
[[4], [4], [4], [4], [4]]

Why are all the elements [4]? Shouldn't it be [[0], [1], [2], [3], [4]]?

Aran-Fey
  • 39,665
  • 11
  • 104
  • 149
piggs_boson
  • 987
  • 2
  • 12
  • 25

2 Answers2

12

You are reusing the same list object. Your generator returns the one object over and over again, manipulating it as it goes, but any other references to it see those same changes:

>>> r = list(gen())
>>> r
[[4], [4], [4], [4], [4]]
>>> r[0] is r[1]
True
>>> r[0][0] = 42
>>> r
[[42], [42], [42], [42], [42]]

Yield a copy of the list or create a new fresh list object instead of manipulating one.

def gen_copy():
    state = [None]

    for i in range(5):
        state[0] = i
        yield state.copy()  # <- copy

def gen_new():
    for i in range(5):
        state = [i]  # <- new list object every iteration
        yield state
Aran-Fey
  • 39,665
  • 11
  • 104
  • 149
Martijn Pieters
  • 1,048,767
  • 296
  • 4,058
  • 3,343
  • Any clue why https://youtu.be/QksUFT2Cmlo?list=PLhTjy8cBISEqkN-5Ku_kXG4QW33sxQo0t&t=305 is not facing the same issue? I wrote the same code but faced the issue mentioned in the question... – Parth Kapadia Dec 02 '22 at 10:08
  • @ParthKapadia scrapy almost certainly clones the item each time it’s yielded. – Martijn Pieters Dec 02 '22 at 22:16
  • @ParthKapadia scrapy even [defines copy methods on the Item class](https://github.com/scrapy/scrapy/blob/master/scrapy/item.py#L114-L120) for this. – Martijn Pieters Dec 02 '22 at 22:19
5

You are yielding the same list/object so you always see the last values added to the list. You should yield a copy:

yield state.copy()

Or create the list inside the first loop:

for i in range(5):
    state = [i]

It would be as easy to create a new list/object each time:

def gen():
    for i in range(5):
        state = [None]
        state[0] = i
        yield state
Aran-Fey
  • 39,665
  • 11
  • 104
  • 149
Padraic Cunningham
  • 176,452
  • 29
  • 245
  • 321