0

I couldn't find it in the runtime, but I am wondering how go decides how much memory is likely to be needed when reallocating maps or slices. So when we look at this (same would apply to slices):

test := map[int]string{}
for i := 0; i < 1000; i++ {
  test[i] = fmt.Sprintf("test-%d", i)
}

How much memory will be allocated at first and how many reallocations take place during the loop?

I am asking this because I am trying to figure out if there are cases where it makes sense to apply a custom reallocation strategy (i.e. use an array instead of a slice and grow the array when needed by a factor which likely meets my application's requirements).

tworabbits
  • 1,203
  • 12
  • 17
  • 1
    You can "preallocate" a map with big capacity with `make(map[int]string, 1000)` for 1000 elements, so no reallocation will occur. – icza Apr 11 '18 at 08:06
  • Yepp that's right, but what if I can't? There might be cases where I can't predict the total length, but I might know in which dimensions it will grow when it exceeds the capacity. – tworabbits Apr 11 '18 at 08:09
  • Even if you figure out how it works in your configuration, it's best not to rely on such internals as it may vary based on what compiler is used, version, OS and architecture. To get a glimpse of how to "measure" it, this might be useful: [How much memory do golang maps reserve?](https://stackoverflow.com/questions/46278003/how-much-memory-do-golang-maps-reserve/46278240#46278240) – icza Apr 11 '18 at 08:28
  • This is an implementation detail, and therefore can/does vary between platforms, compilers, and versions. – Jonathan Hall Apr 11 '18 at 09:07
  • [`append` and `runtime.growslice`](https://stackoverflow.com/a/31790816/720999) — still valid for 1.8; did not check for later versions. – kostix Apr 11 '18 at 09:49
  • The problem with finding the actual runtime code is two-fold: 1) bits of it may be coded in the platform-specific assembly; 2) bits of it may be coded in Go but on the level which is close to the [SSA engine](https://golang.org/doc/go1.8#compiler), so the location and the code are a bit weird—when looked at from the outside ;-) – kostix Apr 11 '18 at 09:51
  • @Flimzy: I am not planning to fiddle around with the internals... I can still grow slices and maps before the runtime would do the job for me when hit the capacity and keep inserting. – tworabbits Apr 11 '18 at 10:15
  • @tworabbits: I wasn't suggesting that you would "fiddle with internals"--only that the answer to your question is not universal. – Jonathan Hall Apr 11 '18 at 10:25

1 Answers1

1

The gist of it is that roughly speaking and barring edge cases, the capacity is usually doubled and there is no control over it. You can implement your own grow/copy mechanism if you need to. You can read the source code here: https://go.googlesource.com/go/+/master/src/runtime/slice.go#89

Not_a_Golfer
  • 47,012
  • 14
  • 126
  • 92
  • Thanks for your answer, the link helps. I was specifically looking for [L112](https://go.googlesource.com/go/+/master/src/runtime/slice.go#112). Looks like size is doubled for capacity < 1024 and factor 1.25 for capacity > 1024. – tworabbits Apr 11 '18 at 10:11
  • Not sure if I wasn't clear in my question, but wouldn't it be possible to get some sort of control by allocating a new map (with lets say double size) when hitting the map capacity and then copying the old into the new map? – tworabbits Apr 11 '18 at 10:19
  • @tworabbits yes it's possible but since you don't have access to the underlying constructs of the map, probably copying over using the exposed API will be so slow that you're better off without it. – Not_a_Golfer Apr 11 '18 at 10:25