Python "restart" generator when used with itertools.cycle or similar construct?

Question

I'd like to yield random values from a generator, and I want to be able to yield them an indefinite number of times, so I wrap the generator in itertools.cycle.

Below is a sample implementation.

from itertools import cycle
import numpy as np

def generator(batch_size):
  for _ in range(batch_size):
    yield np.random.randint(0,50)

np.random.seed(111)
batch_size = 3
gen = cycle(generator(batch_size))

for _ in range(10):
  print([next(gen) for _ in range(batch_size)])

And the output:

[20, 44, 20]
[20, 44, 20]
[20, 44, 20]
[20, 44, 20]
[20, 44, 20]
[20, 44, 20]
[20, 44, 20]
[20, 44, 20]
[20, 44, 20]
[20, 44, 20]

Notice the output just repeats the same three numbers.

My intuition was that the generator should be "re-started" at the beginning of each cycle, but it appears that's not the case. Instead, it appears that the generator is run once to yield the initial list, and then the output is repeated indefinitely.

An alternative strategy to achieve the desired behavior is to discard the cycle and instead pass an infinite=True parameter to the generator, then use a while True loop instead of a for _ in range(...) loop:

def generator(batch_size, infinite):
  while True:
    for _ in range(batch_size):
      yield np.random.randint(0,50)
    if not infinite:
      break

But that feels clumsy.

Is there any construct that I can use to wrap the generator so that it is actually re-started for every cycle?

Could you explain why you're trying to build an infinite stream of random numbers from a function that produces batches of them, rather than from some other primitive (like a function that produces one at a time)? — user2357112, Mar 07 '17 at 19:14
See [Why does Python's itertools.cycle need to create a copy of the iterable?](http://stackoverflow.com/questions/16638639/why-does-pythons-itertools-cycle-need-to-create-a-copy-of-the-iterable) and also the documentation for [**`itertools.cycle`**](https://docs.python.org/2/library/itertools.html#itertools.cycle) — Peter Wood, Mar 07 '17 at 19:18
@user2357112 Sure, I'm randomly sampling windows from images to train a classifier. Specifically using the keras `fit_generator` method. — Alex Klibisz, Mar 07 '17 at 19:19
*'just repeats the same three numbers.'* that's what cycle means. — Peter Wood, Mar 07 '17 at 19:20
@PeterWood this is true. I changed the title to reflect that I'm interested in finding a similar construct that would restart the generator. — Alex Klibisz, Mar 07 '17 at 19:22
Create a new generator each cycle. Better yet, `random.sample(your_seq_of_numbers, batch_size)`. — 9000, Mar 07 '17 at 19:28

score 3 · Accepted Answer · answered Mar 07 '17 at 19:26

Build and flatten an infinite stream of generators:

from itertools import chain, repeat
infinite_gen = chain.from_iterable(map(generator, repeat(batch_size)))
#                 flatten            infinite stream of generators

Or on Python 2:

from itertools import chain, repeat, imap
infinite_gen = chain.from_iterable(imap(generator, repeat(batch_size)))

Python "restart" generator when used with itertools.cycle or similar construct?

1 Answers1