Difference between list comprehension and generator comprehension with `yield` inside

Question

What is the difference between list comprehensions and generator comprehensions with yield inside? Both return a generator object (listcomp and genexpr respectively), but upon full evaluation the latter adds what seem to be rather superfluous Nones.

>>> list([(yield from a) for a in zip("abcde", itertools.cycle("12"))])
['a', '1', 'b', '2', 'c', '1', 'd', '2', 'e', '1']

>>> list(((yield from a) for a in zip("abcde", itertools.cycle("12"))))
['a', '1', None, 'b', '2', None, 'c', '1', None, 'd', '2', None, 'e', '1', None]

How come? What is the scientific explanation?

Answer 1

TLDR: A generator expression uses an implicit yield, which returns None from the yield from expression.

There are actually two things behaving differently here. Your list comprehension is actually thrown away...

• Once again with clarity

Understanding this is easiest if you transform the expressions to equivalent functions. For clarity, let's write this out:

listcomp = [<expr> for a in b]
def listfunc():
    result = []
    for a in b:
        result.append(<expr>)
    return result

gencomp = (<expr> for a in b)
def genfunc():
    for a in b:
        yield <expr>

To replicate the initial expressions, the key is to replace <expr> with (yield from a). This is a simple textual replacement:

def listfunc():
    result = []
    for a in b:
        result.append((yield from a))
    return result

def genfunc():
    for a in b:
        yield (yield from a)

With b = ((1,), (2,)), we would expect the output 1, 2. Indeed, both replicate the output of their respective expression/comprehension forms.

As explained elsewhere, yield (yield from a) should make you suspicious. However, result.append((yield from a)) should make you cringe...

• Yielding the answer

Let's look at the generator first. Another rewrite makes it obvious what is going on:

def genfunc():
    for a in b:
        result = (yield from a)
        yield result

For this to be valid, result must have a value - namely None. The generator does not yield the (yield from a) expression, but its result. You only get the content of a as a side effect of evaluating the expression.

• Returning to the question

If you check the type of your "list comprehension", it is not list - it is generator. <listcomp> is just its name. Yes, that's not a Moon, that's a fully functional generator.

Remember how our transformation put a yield from inside a function? Yepp, that is how you define a generator! Here is our function version, this time with print sprinkled on it:

def listfunc():
    result = []
    for a in b:
        result.append((yield from a))
        print(result[-1])
    print(result)
    return result

Evaluating list(listfunc()) prints None, None (from the append), and [None, None] (from the result) and yields 1, 2. Your actual list contains those None that sneaked into the generator as well! However, it is thrown away and the result is again just a side effect. This is what actually happens:

• A generator is created upon evaluating the list comprehension/listfunc.
• Feeding it to list iterates over it...
  • yield from a yields the values of a to list and returns None to the comprehension/listfunc
  • None is stored in the result list

• At the end of the iteration...

  • return raises StopIteration with a value of [None, None]
  • The list constructor ignores this and throws the value away

• Moral of this story

Don't use yield from inside of comprehensions. It does not do what you think it does.

Answer 2

The value of the yield from expression is None. The fact that your second example is a generator expression means that it is already implicitly yielding from the iterator, so it will also yield the value of the yield from expression. See this for a more detailed answer.

Answer 3

Both examples are deprecated in Python 3.8 because of the confusion and throw SyntaxError: 'yield' inside list comprehension. See the buglog for 3.8 for the release notes.