Does Go have no real way to shrink a slice? Is that an issue?

Question

I've been trying out Go for some time and this question keeps bugging me. Say I build up a somewhat large dataset in a slice (say, 10 million int64s).

package main

import (
    "math"
    "fmt"
)

func main() {
    var a []int64
    var i int64;
    upto := int64(math.Pow10(7))
    for i = 0; i < upto; i++ {
        a = append(a, i)
    }
    fmt.Println(cap(a))
}

But then I decide I don't want most of them so I want to end up with a slice of just 10 of those. I've tried both slicing and delete techniques on Go's wiki but none of them seem to reduce the slice's capacity.

So that's my question: does Go has no real way of shrinking the capacity of a slice that would be similar to realloc()-ing with a smaller size argument than in your previous call on the same pointer in C? Is that an issue and how should one deal with it?

Answer 1

To perform an, in effect, a realloc of a slice:

a = append([]T(nil), a[:newSize]...) // Thanks to @Dijkstra for pointing out the missing ellipsis.

If it does a copy of newSize elements to a new memory place or if it does an actual in place resize as in realloc(3) is at complete discretion of the compiler. You might want to investigate the current state and perhaps raise an issue if there's a room for improvement in this.

However, this is likely a micro-optimization. The first source of performance enhancements lies almost always in selecting a better algorithm and/or a better data structure. Using a hugely sized vector to finally keep a few items only is probably not the best option wrt to memory consumption.

EDIT: The above is only partially correct. The compiler cannot, in the general case, derive if there are other pointers to the slice's backing array. Thus the realloc is not applicable. The above snippet is actually guaranteed to peform a copy of 'newSize' elements. Sorry for any confusion possibly created.

Answer 2

Go does not have a way of shrinking slices. This isn't a problem in most cases, but if you profile your memory use and find you're using too much, you can do something about it:

Firstly, you can just create a slice of the size you need and copy your data into it. The garbage collector will then free the large slice. Copy built-in

Secondly, you could re-use the big slice each time you wish to generate it, so you never allocate it more than once.

On a final note, you can use 1e7 instead of math.Pow10(7).

Answer 3

Let's see this example:

func main() {
    s := []string{"A", "B", "C", "D", "E", "F", "G", "H"}
    fmt.Println(s, len(s), cap(s))  // slice, length, capacity

    t := s[2:4]
    fmt.Println(t, len(t), cap(t))

    u := make([]string, len(t))
    copy(u, t)
    fmt.Println(u, len(u), cap(u))
}

It produces the following output:

[A B C D E F G H] 8 8
[C D] 2 6
[C D] 2 2

s is a slice that holds 8 pieces of strings. t is a slice that keeps the part [C D]. The length of t is 2, but since it uses the same hidden array of s, its capacity is 6 (from "C" to "H"). The question is: how to have a slice of [C D] that is independent from the hidden array of s? Simply create a new slice of strings with length 2 (slice u) and copy the content of t to u. u's underlying hidden array is different from the hidden array of s.

The initial problem was this: you have a big slice and you create a new smaller slice on it. Since the smaller slice uses the same hidden array, the garbage collector won't delete the hidden array.

See the bottom of this post for more info: http://blog.golang.org/go-slices-usage-and-internals .

Answer 4

Additionally you can re-use most of the allocated memory during work of yours app, take a look at: bufs package

PS if you re-alocate new memory for smaller slice, old memory may not be freed in same time, it will be freed when garbage collector decides to.

Answer 5

You can do that by re-assigning the slice's value to a portion of itself

  a := []int{1,2,3}
  fmt.Println(len(a), a) // 3 [1 2 3]

  a = a[:len(a)-1]
  fmt.Println(len(a), a) //2 [1 2]

Answer 6

There is a new feature called 3-index slice in Go 1.2, which means to get part of a slice in this way:

slice[a:b:c]

In which the len for the returned slice whould be b-a, and the cap for the new slice would be c-a.

Tips: no copy is down in the whole process, it only returns a new slice which points to &slice[a] and has the len as b-a and cav as c-a.

And that's the only thing you have to do:

slice= slice[0:len(slice):len(slice)];

Then the cap of the slice would be changed to len(slice) - 0, which is the same as the len of it, and no copy is done.