0

It is easy to convert an entire iterator sequence into a list using list(iterator), but what is the best/fastest way to directly create a sublist from an iterator without first creating the entire list, i.e. how to best create list(iterator)[m:n] without first creating the entire list?

It seems obvious that it should not* (at least not always) be possible to do so directly for m > 0, but it should be for n less than the length of the sequence. [p for i,p in zip(range(n), iterator)] comes to mind, but is that the best way?

The context is simple: Creating the entire list would cause a RAM overflow, so it needs to be broken down. So how do you do this efficiently and/or python-ic-ly?

*The list comprehension I mentioned could obviously be used for m > 0 by calling next(iterator) m times prior to execution, but I don't enjoy the lack of python-ness here.

MrArsGravis
  • 568
  • 1
  • 4
  • 9

1 Answers1

1

itertools.islice:

from itertools import islice

itr = (i for i in range(10))
m, n = 3, 8
result = list(islice(itr, m, n))
print(result)
# [3, 4, 5, 6, 7]

In addition, you can add an argument as the step if you wanted:

itr = (i for i in range(10))
m, n, step = 3, 8, 2
result = list(islice(itr, m, n, step))
print(result)
# [3, 5, 7]
iz_
  • 15,923
  • 3
  • 25
  • 40
  • Yup: For `m=5000` and `n=6000`, i.e. relatively small numbers, this method was indeed about 40% faster than mine. Unfortunately I'll have to restructure my original problem, because there's no way to handle even chunks of 1e15 elements -- but that's a different issue. – MrArsGravis Jan 16 '20 at 02:56
  • @MrArsGravis What exactly are the issues you are facing? Does my answer solve your question? – iz_ Jan 16 '20 at 23:10
  • Yes, you posted a solution, and one that's faster than the one I proposed. I thought I would keep the question open in case someone else has a different suggestion, but that doesn't really seem to be the case. As for what I was originally trying to solve: It just takes too plain long to iterate through large chunks of the sequence, so I resorted to solving it differently, but I think this question still has merit of its own... – MrArsGravis Jan 17 '20 at 00:56
  • I see. That makes sense, but in my opinion you can't get faster than this. I would be surprised if there was something that could beat a standard library function that was designed exactly for this purpose. – iz_ Jan 17 '20 at 01:40