1

To amortize the function call overhead I have changed my generator so that it yields a fixed-length list of a few values instead of yielding one value at a time. The generator, as it originally stood, unpickled an object from a file that contains several pickled objects, and yielded it. These were then processed inside a for loop that consumed the generator. This approach turned out to be a lot slower than having the object processing code inside a hand unrolled loop that unpickled several consecutive items in the file at a time. I am attempting a compromise. My modified generator yields a fixed-length list of pickled objects at a time. I am looking for a Pythonic way to unpack that packet of unpickled objects at the consumer side.

Is there a way to deconstruct those packets without having an extra nested loop ? I incorrectly assumed that the * operator will do it like so:

for x in *packetizing_generator(): f(x)

Nested loop of course works, but am wondering if there is a shorter and more elegant way.

san
  • 4,144
  • 6
  • 32
  • 50

1 Answers1

1

I think this is what you are talking about, let me know if I'm not on the right track.

for a, b, c in your_generator:
    # do stuff

Assumes your_generator yields results as a sequence of exactly 3 items.

You can construct a generator that yields exactly n items at a time to use in your for loop with the following method:

itertools.izip_longest(*[your_generator]*n)

And if your generator does not yield a single item at a time, and you want to change the number of items it yields you use the following:

itertools.izip_longest(*[itertools.chain(*your_generator)]*n)

Here is an example:

>>> from itertools import izip_longest, chain, combinations
>>> for a, b, c, d in izip_longest(*[chain(*combinations(range(4), 2))]*4):
...     print a, b, c, d
... 
0 1 0 2
0 3 1 2
1 3 2 3

This takes a generator that originally yielded items two at a time, and instead yields elements four at a time.

Andrew Clark
  • 202,379
  • 35
  • 273
  • 306
  • So the generator needs to be given a yield length parameter? You should definitely be able to write your generator in such a way that it always yields a specific number of items. – Andrew Clark Apr 01 '11 at 20:57
  • Yeah. the generator is written and done with. At the consumer side I have a nested loop. Was wondering if there is Pythonic way of deconstructing that packet – san Apr 01 '11 at 21:00
  • I think nested loops probably is the way to go, but you might get better performance from using `chain.from_iterable(your_generator)` instead of `chain(*your_generator)`, the use of the `*` there forces the entire generator to be consumed which is unnecessary. – Andrew Clark Apr 01 '11 at 21:42
  • Thanks for your answer, this is a typo corrected version of a previous comment. The nested for loop turned out to be the fastest. But good to know that the "*" operation works only in the context of a function call. So its bit of a bummer because its function calls that I was hoping to avoid, using something like for x in *my_generator(): – san Apr 01 '11 at 21:45