10

I'm using a std::deque to store a fairly large number of objects. If I remove a bunch of those objects, it appears to me that its memory usage does not decrease, in a similar fashion to std::vector.

Is there a way to reduce it? I know that in a vector you have to use the 'swap trick', which I assume would work here too, but I'd rather avoid that since it would require copying all the elements left in the container (and thus requires that you have enough memory to store every object twice). I'm not intimately familiar with the implementation of deque, but my understanding of it is that it might be possible to achieve such a thing without lots of copies (whereas with a vector it's clearly not).

I'm using the VC++ (Dinkumware) STL, if that makes any difference.

Peter
  • 7,216
  • 2
  • 34
  • 46
  • 2
    Have you established that your deque implementation isn't already freeing blocks of memory as soon as enough items are removed to empty them? Or is it that you really want to squeeze the last few bytes out by reallocating the block at each end? – Steve Jessop Aug 07 '09 at 01:01
  • I think so, in a fairly rough and ready way: I add 100,000 items -> Memory usage is ~90MB. I add another 100,000 -> memory usage is ~170MB. I remove 100,000 items -> memory usage is still ~170MB. Add another 100,000 -> still 170. I assume 100,000 items is more than enough that it would have empty blocks which it would have freed if it were going to. – Peter Aug 07 '09 at 01:11
  • 6
    Memory usage of the process, or memory usage of the collection? Just because the collection frees the memory doesn't mean it goes back to the OS, so try allocating 80MB of char array after removing the items, and see whether usage goes to 250MB, or stays at 170. Apologies if you already know all this stuff and have accounted for it - millions wouldn't. – Steve Jessop Aug 07 '09 at 01:16
  • Also note that I genuinely don't know what to expect - I just have no idea how dinkum in particular or libraries in general manage deque memory. As soon as someone comes along who does, you can safely ignore me :-) – Steve Jessop Aug 07 '09 at 01:19
  • All good questions - my numbers were the process usage, but it seems I was correct nonetheless. The 80MB of char array does push the memory up to 250MB. – Peter Aug 07 '09 at 01:28
  • 1
    Just noticed that a single 80MB char array is over-simplistic, since even if the deque did free the memory, it might have left free space fragmented. So my proposed test still doesn't prove the deque is hanging on to the memory, sorry. Was quite late at night. You'd have to make a large total of fairly small allocations to prove there's no way your app can make use of the slack. Maybe use a second deque rather than plain arrays, to get a comparable allocation pattern. Or check how much of dinkum's implementation is visible in the headers. – Steve Jessop Aug 07 '09 at 10:00

4 Answers4

15

There is no way to do this directly in a std::deque. However, it's easy to do by using a temporary (which is basically what happens in a std::vector when you shrink it's capacity).

Here is a good article on std::deque, comparing it to std::vector. The very bottom shows a clean way to swap out and shrink a vector, which works the same with deque.

Reed Copsey
  • 554,122
  • 78
  • 1,158
  • 1,373
  • Thanks. I'd suspected that there might not be any way to do it, but it seems like it should be possible... I explained in the question why I'm loath to use the swap thing in this case, but maybe I'll have to look into it regardless. – Peter Aug 07 '09 at 02:47
  • Peter - the best option with the "swap" thing is to use heap allocated elements. In that case, you're only making a copy of references, instead of a copy of all of the elements plus their memory. That is pretty minor, typically. – Reed Copsey Aug 07 '09 at 16:47
5

As added information to this:

In C++0x/C++11, deque (and several other containers) has a new function called "shrink_to_fit" which will remove the excess items and basically align capacity() == size()

David
  • 3,324
  • 2
  • 27
  • 31
  • 2
    capacity() still can still be higher than size() after shrink_to_fit() - it is a non-binding request; `void shrink_to_fit(){}` is a perfectly legal implementation, actually. But even with a good implementation, capacity() can be higher than size(), for instance if the container has too few elements, it might still worth to allocate more memory. (Think of empty containers.) – Lrdx Oct 27 '15 at 23:14
4

The memory size of a deque might or might not shrink. When and How this happens is implementation specific. Unfortunately you don't have much manual control over this since deques lack even capacity() or reserve().

I'd suggest swap() if you indeed detect memory release isn't being performed at your convenience.

An intimate knowledge of deque memory management may probably be gained from Dikum website (this is your current implementation, right?)

Alexandre Bell
  • 3,141
  • 3
  • 30
  • 43
1

std::deque will return memory to its allocator. Often this allocator won't return the memory to the OS. In such cases, it appears as if memory is not "released". Good memory leak detectors will be satisfied as soon as memory is returned to the allocator, and understand that not all memory is released by free().

MSalters
  • 173,980
  • 10
  • 155
  • 350