0

I want to divide an iterator into N equal chunks. The chunks themselves need to be iterators.

Here's some examples of what I want to achieve:

iter1 = iter(range(10))
chunks = split_n(iter1, n=2)

outputs iterator:
[[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9]]

iter2 = iter(range(20))
chunks = split_n(iter2, n=4)

outputs iterator:
[[0, 1, 2, 3, 4],
[5, 6, 7, 8, 9],
[10, 11, 12, 13, 14],
[15, 16, 17, 18, 19]]

If an iterator is not evenly divisible, I want the remaining elements to be spread out so that the chunks are "as equal" as possible.

iter3 = iter(range(30))
chunks = split_n(iter3, n=4)

outputs iterator: 
[[0, 1, 2, 3, 4, 5, 6, 7],
[8, 9, 10, 11, 12, 13, 14, 15], 
[16, 17, 18, 19, 20, 21, 22],
[23, 24, 25, 26, 27, 28, 29]]

One important caveat is that at no point should an iterator be converted to a list, as the iterators can be very huge and it would simply consume too much memory.

Thegerdfather
  • 398
  • 4
  • 13
  • What should be the result of `split_n(itertools.count(), 2)`? – mkrieger1 Dec 27 '22 at 01:25
  • 5
    It's not possible unless the number of elements is known in advance, and then it's already answered here: [Iterate an iterator by chunks (of n) in Python?](https://stackoverflow.com/questions/8991506/iterate-an-iterator-by-chunks-of-n-in-python) – mkrieger1 Dec 27 '22 at 01:32
  • @mkrieger1 That doesn't really answer my question, as the linked question is about dividing an iterator into chunks of N length. My question is specific to iterators divided into N chunks. – Thegerdfather Dec 27 '22 at 01:55
  • Okay, then can you please answer my first comment? – mkrieger1 Dec 27 '22 at 01:55
  • The input can't be an infinite iterator. It would produce an error – Thegerdfather Dec 27 '22 at 02:06
  • 1
    Okay, so the input is a finite iterator with length M. Then you apply the solutions from the linked question where you divide it into N chunks of size M/N. – mkrieger1 Dec 27 '22 at 02:08

0 Answers0