3

PEP 380 mentions that the yield from expr syntax can be optimized in Python.

PEP 380 - Optimizations

Using a specialised syntax opens up possibilities for optimisation when there is a long chain of generators. Such chains can arise, for instance, when recursively traversing a tree structure. The overhead of passing __next__() calls and yielded values down and up the chain can cause what ought to be an O(n) operation to become, in the worst case, O(n**2).

A possible strategy is to add a slot to generator objects to hold a generator being delegated to. When a __next__() or send() call is made on the generator, this slot is checked first, and if it is nonempty, the generator that it references is resumed instead. If it raises StopIteration, the slot is cleared and the main generator is resumed.

This would reduce the delegation overhead to a chain of C function calls involving no Python code execution. A possible enhancement would be to traverse the whole chain of generators in a loop and directly resume the one at the end, although the handling of StopIteration is more complicated then.

Does CPython implement this optimization?

Community
  • 1
  • 1
Paebbels
  • 15,573
  • 13
  • 70
  • 139
  • You should specify the location of and then quote the specific line or lines you are referring to. Since the PEP was written and approved by CPython developers and first implemented for CPython, I would guess 'yes', whatever you are referring to, but I would not know for sure. You could write timing tests, or inquire on python-list to get a better answer. The author, Greg Ewing, occasionally posts there. – Terry Jan Reedy Apr 24 '17 at 16:06

1 Answers1

2

Doesn't look like it. As of Python 3.6, the generator object structure doesn't have the proposed field, and the code path for yield from through a chain of generators always goes through the process of resuming their Python stack frames individually. (This code path goes from the YIELD_FROM opcode through _PyGen_Send/gen_iternext to gen_send_ex, and from there to PyEval_EvalFrameEx and on to the YIELD_FROM of the next generator in the chain.)

user2357112
  • 260,549
  • 28
  • 431
  • 505