Why does adding parenthesis around a yield call in a generator allow it to compile/run?

Question

I have a method:

@gen.coroutine
def my_func(x):
    return 2 * x

basically, a tornado coroutine.

I am making a list such as:

my_funcs = []
for x in range(0, 10):
    f = yield my_func(x)
    my_funcs.append(x)

In trying to make this a list comprehension such as:

my_funcs = [yield my_func(i) for i in range(0,10)]

I realized this was invalid syntax. It turns out you can do this using () around the yield:

my_funcs = [(yield my_func(i)) for i in range(0,10)]

Does this behavior (the syntax for wrapping a yield foo() call in () such as (yield foo() ) in order to allow this above code to execute) have a specific type of name?
Is it some form of operator precedence with yield?
Is this behavior with yield documented somewhere?

Python 2.7.11 on OSX. This code does need to work in both Python2/3 which is why the above list comprehension is not a good idea (see here for why, the above list comp works in Python 2.7 but is broken in Python 3).

@MartijnPieters but `yield` is [not in the Python2 docs under operator precedence](https://docs.python.org/2/reference/expressions.html#operator-precedence) - code like `m = [foobar() for i in range(0, 10)]` works fine and as expected. — enderland, Oct 17 '16 at 19:51
@PadraicCunningham I added an example of why I ended up with list comprehension in the first place. — enderland, Oct 17 '16 at 19:54
@enderland: which is definitely an oversight, because just like `lambda`, `yield` is an expression atom and has a precedence. It looks like it has a lower precedence even than `lambda`. — Martijn Pieters, Oct 17 '16 at 19:57
Not sure how you end up with None, are you returning the list in a function? — Padraic Cunningham, Oct 17 '16 at 19:57
@PadraicCunningham: In Python 3, a list comp is given a new scope (a hidden function basically), using `yield` in it produces a generator function, and because nothing is sending anything to the generator, every `yield` expression defaults to producing `None`. This is a know issue with using `yield` in generator expressions and the other comprehensions, as well as list comps in Python 3. — Martijn Pieters, Oct 17 '16 at 19:58
Side note: *don't use a list comp* for the side effects. You are now building a very expensive and useless list of `None` references. — Martijn Pieters, Oct 17 '16 at 20:00
@MartijnPieters I am not planning on using list comprehension here due to those issues (which are discussed elsewhere here). But my curiosity about _why_ got me going down the rabbit trail, so to speak ;-) — enderland, Oct 17 '16 at 20:01
"While this compiles and runs, it adds None into it because of how yield works in list comprehension." - at least on Python 2, putting the `yield` inside a list comprehension shouldn't have had such effect. It would have had a different, even crazier effect in Python 3, or with something like a dict or set comprehension, but not a Python 2 list comprehension. — user2357112, Oct 17 '16 at 20:02
@user2357112 I just removed that part of the question, I'm more curious about the `()` than the actual explanation for the generator/yield stuff. That is discussed [elsewhere in depth](http://stackoverflow.com/a/32139977/1048539). — enderland, Oct 17 '16 at 20:03
@MartijnPieters but why does it result in *alternating* values and None? for example: `list((yield i) for i in range(2))` results in `[0, None, 1, None]`. I guess it's time for me to finally read PEP 342... — juanpa.arrivillaga, Oct 17 '16 at 20:06
@juanpa.arrivillaga see [this answer](http://stackoverflow.com/questions/32139885/yield-in-list-comprehensions-and-generator-expressions/32139977#32139977). Incidentally, this code does need to work in both Python2.7 and Python3, so I'm most _definitely_ not going to try to use a list comp to create this. — enderland, Oct 17 '16 at 20:06
@PadraicCunningham: yeah, the claim was puzzling, and I don't have the time and energy right now to reproduce what happens here. — Martijn Pieters, Oct 17 '16 at 20:08

score 5 · Accepted Answer · edited Oct 17 '16 at 20:14

yield expressions must be parenthesized in any context except as an entire statement or as the right-hand side of an assignment:

# If your code doesn't look like this, you need parentheses:
yield x
y = yield x

This is stated in the PEP that introduced yield expressions (as opposed to yield statements), and it's implied by the contexts in which yield_expr appears in the grammar, although no one is expecting you to read the grammar:

A yield-expression must always be parenthesized except when it occurs at the top-level expression on the right-hand side of an assignment.

Why does adding parenthesis around a yield call in a generator allow it to compile/run?

1 Answers1