26

I have a deque object what holds a large amount of data. I want to extract, say, 4096 elements from the front of the queue (I'm using it as a kind of FIFO). It seems like there should be way of doing this without having to iterate over 4096 pop requests.

Is this correct/efficient/stupid?

A = arange(100000)
B = deque()
C = [] # List will do 
B.extend(A) # Nice large deque

# extract 4096 elements
for i in xrange(4096):
   C.append(A.popleft())
alexwlchan
  • 5,699
  • 7
  • 38
  • 49
Ross W
  • 1,300
  • 3
  • 14
  • 24
  • 6
    Yes, it is correct. Yes, it is reasonably efficient though it can be further sped-up with boundmethods and itertools. No, it isn't stupid :-) – Raymond Hettinger Mar 01 '12 at 02:06

2 Answers2

14

There is no multi-pop method for deques. You're welcome to submit a feature request to bugs.python.org and I'll consider adding it.

I don't know the details of your use case, but if your data comes in blocks of 4096, consider storing the blocks in tuples or lists and then adding the blocks to the deque:

block = data[:4096]
d.append(block)
...
someblock = d.popleft()
Raymond Hettinger
  • 216,523
  • 63
  • 388
  • 485
  • 5
    Thanks Raymond, One of the reasons for using the deque is the maxlength parameter - I'm using this as a container for data that is constantly being streamed and hence I can easily control the amount of data kept. The FIFO aspect of it is good too. – Ross W Mar 01 '12 at 14:45
5

Where you're using a deque the .popleft() method is really the best method of getting elements off the front. You can index into it, but index performance degrades toward the middle of the deque (as opposed to a list that has quick indexed access, but slow pops). You could get away with this though (saves a few lines of code):

A = arange(100000)
B = deque(A)
C = [B.popleft() for _i in xrange(4096)]
g.d.d.c
  • 46,865
  • 9
  • 101
  • 111
  • 9
    I like the OP's version a little better. But if there is a need to collapse it to a one-liner, here some fast itertools variants: ``C = map(apply, repeat(B.popleft, 4096))`` or you could try ``C = list(starmap(B.popleft, repeat((), 4096)))`` – Raymond Hettinger Feb 29 '12 at 23:56
  • 3
    @RaymondHettinger: Your commentary here is substantially more useful than [your answer above](https://stackoverflow.com/a/9508505/2809027), which largely obviates the point of using a `deque` in the first place. Unsurprisingly, the former received more upvotes than the latter. *Always check those comments, folks.* – Cecil Curry Aug 28 '18 at 05:59