5

Let's say I have a simple map with string as keytype and a self-defined struct as valuetype. Like this: map[string]*struct

I populate this map with a lot of different values and a lot of these values will never be used again after a certain period of time.

So I wasn't sure whether the golang garbage collector will clean up my map for me or I need to do it myself. Then I came across this answer on a different question: Is it safe to remove selected keys from Golang map within a range loop?

This makes it look like the garbage collector won't do it for me and my only solution is to set my map to nil if i want to free up some memory every now and then.

Is this true? Or is there another way to do it without losing values in my map that are not 'inactive'?

Community
  • 1
  • 1
sieberts
  • 528
  • 6
  • 15
  • 2
    The GC collects all memory that is unreachable. Both maps, and the deleted values from maps are collected. The map structure may or may not be compacted, but that's separate from the values being collected. – JimB Apr 20 '16 at 15:13
  • 1
    I'm not sure exactly what you're asking. The garbage collector won't get rid of entries in your map because they're referenced by the map (assuming that the map itself is reachable). If you have entries in the map that you don't need anymore, you should delete them. If you set the value for a particular key in the map to nil, the struct that that map entry pointed to will be garbage collected, assuming that it's not reachable in some other way. – Andy Schweig Apr 20 '16 at 15:17
  • So what verran suggests about delete not freeing up memory in http://stackoverflow.com/questions/36747776/do-i-need-to-set-a-map-to-nil-in-order-for-it-to-be-garbage-collected is not true? – sieberts Apr 20 '16 at 15:24
  • @siebertschoutteten you just linked back to this question. – hobbs Apr 20 '16 at 15:33
  • 1
    I think you're referring to http://stackoverflow.com/a/23231539/152948 — and the answer is yes and no. The *bucket* is never cleaned up. If the map is storing a value type (like `int`), the storage for that `int` is inside the bucket, so it will also never be cleaned up unless you nil out the map. But if the map is storing pointers, then that means the *storage for the pointer* will persist, but the pointed-to value will be GC'd provided nothing is pointing to it. – hobbs Apr 20 '16 at 15:46
  • Yeah i wanted to refer to that particular answer, my bad! If the storage for the pointers persists, will memory be freed up if I actually set my map to nil? Or will this have the same effect as just deleting them? – sieberts Apr 20 '16 at 15:51
  • @hobbs: even though buckets aren't currently compacted, the memory for large values is still reclaimed. Currently anything over 128 bytes is stored via a pointer even if it's a value type. – JimB Apr 20 '16 at 15:58
  • @siebertschoutteten, yes (as Verran mentioned in his answer) setting the map to nil will allow the GC to reclaim everything. – matt.s Apr 20 '16 at 16:41

2 Answers2

6

To try and answer this fully, we need to figure out what the question is exactly.

For the title question:

Do I need to set a map to nil in order for it to be garbage collected?

No, once the map value is out of scope, it will be garbage collected like any other value.

I populate this [map[string]*stuct] map with a lot of different values and a lot of these values will never be used again after a certain period of time.

This example map you show contains pointer values, and for as long as they are contained in the map, the values to which they point will never be collected. Deleting the values from the map (using delete or setting the key to another value) will allow the memory referenced by the pointers to be collected. There is no special handling that needs to be done around a map to ensure garbage collection.

Now, the internal structures of a map are not currently compacted, and small values (including pointers, and anything under 128 bytes) are stored directly in the hash buckets. A map with millions of entries isn't going to get smaller immediately after deleting those entries, so if you need to free that memory it's best to copy the remaining values you want to a new map. This is analogous to having a large slice that's no longer needed except for a few values, where you need to copy the remaining values to a new slice to free the original backing array.

JimB
  • 104,193
  • 13
  • 262
  • 255
  • 1
    So let me get this right. It will definitely affect my memory usage in a good way if I copy my remaining values to a new map (we are talking milions of entries here) in stead of doing nothing or deleting the entries who are no longer needed? – sieberts Apr 21 '16 at 08:34
  • @siebertschoutteten: if the time trade-off is worth it for you, then yes. The best way to answer that question is to simply test it out and see. – JimB Apr 21 '16 at 12:50
  • That's what I needed to know! – sieberts Apr 21 '16 at 15:03
  • You say that "the internal structure of a map is not compacted". I suppose that means that the memory blocks previously used by now deleted entries are kept around. Are these then reused for new items added to the map? If so, then in most cases it's not going to be a problem to still have them around. Actually, if anything, it's likely to accelerate new insertions since these blocks of memory are readily available. – Alexis Wilke Jul 26 '19 at 06:57
0

You can delete() individual entries from the map, even while iterating over it. If the entry's value is unreachable from anything else (and it is big enough), they will be GC'ed.

The question you looked at referenced some old code (you can look at the source yourself), memory should be emptied after objects are removed from the map.

matt.s
  • 1,698
  • 1
  • 20
  • 29
  • The code for deletion hasn't really changed since that time, other than being translated from C to Go. Which is a big deal, but it still doesn't consolidate buckets. I think Verran was speaking a little bit imprecisely but is still fundamentally correct. – hobbs Apr 20 '16 at 15:48
  • It may be referencing some old code but I can't find any source to prove what you are saying is true? When did they change it? I quickly read the go 1.5 and 1.6 release notes and could'nt find anything about it. – sieberts Apr 20 '16 at 15:50
  • I'm pretty sure memclr in https://github.com/golang/go/blob/master/src/runtime/hashmap.go#L587 will reclaim the memory for values. I think Verran's information is accurate about the keys, but not about the values – matt.s Apr 20 '16 at 16:41
  • @matt.s: `memclr` zeros the bytes for a region of memory, it doesn't reclaim the space for that value. Run this http://play.golang.org/p/6NcaMtFaHy with `GODEBUG=gctrace=1`, and you will see that the collection after deleting all keys does nothing, and the heap isn't completely freed until the map value itself can be collected. – JimB Apr 20 '16 at 19:07
  • @JimB, I'm not sure about that. If you give the GC time/need to work, it looks like it reclaims most of that. If you look at this example http://play.golang.org/p/VanBaBR55K, why does the memory go down? – matt.s Apr 20 '16 at 19:45
  • @matt.s: when you make the `valueSize` larger than 128, it's stored in the hash bucket as a pointer, so the actual value can be GC'ed. – JimB Apr 20 '16 at 20:00
  • @JimB, that's what I was saying; the values can be collected (not necessarily the keys). I'll make that clearer in my answer. – matt.s Apr 20 '16 at 20:04