"TypeError: 'generator' object is not subscriptable" when I try to deal with 2-dimensional list with for-loop

Question

class test(object):
    def __init__(self, name):
        self.name = ''


testList = [(test("empty") for i in range(3)) for j in range(2)]


for m in range(3):
    for n in range(2):

        testList[m][n].name = "changed"

I'm trying to check and change items of a 2-dimensional list which contains objects only. I built 2d list first and tried to affect the items in it with double for-loop but it returns TypeError.

Traceback (most recent call last):
  File "test.py", line 12, in <module>
    testList[m][n].name = "changed"
TypeError: 'generator' object is not subscriptable

I really couldn't understand what's going on here as it seems quite simple and viable. The script could not run with testList[0][0].name = "changed" (instead of testList[m][n]) so I suspect that the loop is not allowed to run like this. But why?

Answer 1

When you type (foo for i in bar) you get a generator, when you type [foo for i in bar] you get a list. The difference between these two is that the generator create elements (generate) as it is traversed, while a list hold all items on memory. This is why (i for i in range(10))[2] is not possible, while [i for i in range(10)][2] is.

You should use generators when the whole set of items are too big to keep in memory or simply when you don't need them all in memory at same time. They are good for traversing files while keeping constant memory usage for example.

Now if you want to subscript something, like foo[some_index] then foo need to be subscriptable and generators aren't because doing so would throw away the whole point of generator existence. While someone may arg that (i for i in range(10))[2] is okay, expanding some generators may end up in infinite loop, for example:

from itertools import count
even = (i for i in count() if i % 2 == 0)

This is perfect valid code. The count() returns an infinite generator. If we can argue that even[1] would be 2 what would be even[-1]? There is no last even number. So the computation would take forever.

Anyway. Generators are common in python and sooner or later you'll need to convert then to a list or a tuple, you can do this by passing they to the list or tuple constructor list(range(10)) or tuple(range(10)).

Now I think that we have enough background to answer your question. You are doing this testList = [(test("empty") for i in range(3)) for j in range(2)] which gives us a list of generators. So testList[m][n] reduces to something like (test("empty") for i in range(3))[n] which is where things blow up.

If you just replace parenthesis by brackets you solve your problem, so testList = [[test("empty") for i in range(3)] for j in range(2)].

Answer 2

You didn't create a list of lists, you created a list of generator objects. Those generator objects are lying dormant, they are not active until code iterates over them. Even then, you don't have a sequence, which is what is required to use indexing. To be able to assign to indexes you need a mutable sequence.

If you want to make each nested index mutable, you have to generate lists, not generators. Replace the (...) parentheses with [...] square brackets to create a list comprehension instead:

testList = [[test("empty") for i in range(3)] for j in range(2)]

List comprehensions are executed immediately, there and then, producing a list object. Lists are mutable sequences.