0

for my last project in Python3, i used a custom lazy generator to generate my data. Then use imap from a Pool (multiprocessing). So at this point, not any computation have been made. The next step is to output the computed data on a file. To do so, I either print(list(data)) or print(*data) which causes the computation of the whole data (approx 1.5Gib right now, bit will grow fast), either do a for loop and print each piece of data, which do a lot of call to print (approx 10e6 calls right now, but will grow fast).

So, is there a way to make print iterate over a lazy generator?

Thank you.

Paul
  • 315
  • 1
  • 9
  • 1
    Just call `print` in a loop. I believe the function call overhead will be the least of your worries. – Zulan May 20 '16 at 19:47
  • 1
    If `g` is a generator then the code `for item in g: print(item)` *is* code which iterates `print` over a lazy generator. – John Coleman May 20 '16 at 19:54
  • that's the main lead, but i don't find it _nice_. Thus, whith a flush=False, this could be the best effective solution. – Paul May 20 '16 at 19:54

1 Answers1

0

Using this recipe from the itertools docs:

def grouper(iterable, n, fillvalue=None):
    "Collect data into fixed-length chunks or blocks"
    # grouper('ABCDEFG', 3, 'x') --> ABC DEF Gxx
    args = [iter(iterable)] * n
    return izip_longest(fillvalue=fillvalue, *args)

You can batch the calls to print yourself:

for batch in grouper(data, 1000, ''):
    print('\n'.join(batch))
Alex Hall
  • 34,833
  • 5
  • 57
  • 89