Why is this hashmap with initial capacity trying to resize?

Question

Why is this code throwing exception?

public static void main(String[] args) {
    Map<Integer, Integer> map = new HashMap<>(Integer.MAX_VALUE);
    System.out.println("map size: "+map.size());
    map.put(1, 1);
    System.out.println("map size: "+map.size());
}

Output:

map size: 0
Exception in thread "main" java.lang.OutOfMemoryError: Java heap space
    at java.util.HashMap.resize(HashMap.java:703)
    at java.util.HashMap.putVal(HashMap.java:628)
    at java.util.HashMap.put(HashMap.java:611)
    at com.fredcrs.codejam.NumberToBinary.main(NumberToBinary.java:24)

Shouldnt the hashmap only resize to a bigger one when it is full ?

EDIT: It also throw the same exception when initializing it with:

Map<Integer, Integer> map = new HashMap<>(Integer.MAX_VALUE-3);

But you are telling it to have an `initialCapacity` of MAX_VALUE see the [javadocs](https://docs.oracle.com/javase/7/docs/api/java/util/HashMap.html#HashMap(int)) — Scary Wombat, Aug 07 '17 at 02:24
yes, but initial capacity means the size of the array used to index the key (hashed) — fredcrs, Aug 07 '17 at 02:26

Jim Garrison · Accepted Answer · 2017-08-07T03:05:21.160

3

new HashMap<>(Integer.MAX_VALUE);

You are asking for an initial array size of 2³¹-1 elements, or 2,147,483,647. At 8 bytes per element (a reference is 64 bits) that's about 16 GB of memory.

Unless you have 18GB or so available to the heap you will always get a OOM error.

You asked for 16GB worth of array memory, and it's going to fail unless that memory is available. Whether it fails at instantiation or first insert is an implementation detail. At some point in the past it would fail on instantiation. More recently the code was changed to wait until the first insert. That change is possible because the detail of when the array is allocated is not part of any external contract -- i.e. it is not mentioned in the JavaDoc.

edited Aug 07 '17 at 03:05

answered Aug 07 '17 at 02:28

Jim Garrison

85,615
20
155
190

Ok, so my guess is that when I create the hashmap object, it does not allocate the array...it does only when adding the first element – fredcrs Aug 07 '17 at 02:31
1

every time an element is `put` the capacity needs to be checked and maybe incremented, so I guess the Author decoded not to allocate memory initially – Scary Wombat Aug 07 '17 at 02:32
2

@fredcrs - `HashMap` used to allocate the array initially, but now it is allocated on first insert, which is a pretty reasonable and powerful optimization when you have many empty maps. – BeeOnRope Aug 07 '17 at 02:41
@JimGarrison the *actual* max size is `MAXIMUM_CAPACITY = 1 << 30`; that's 2 pow 30 != Integer.MAX_VALUE – Eugene Aug 07 '17 at 11:01
@JimGarrison - due to padding and the extra `mark` word (see this : https://stackoverflow.com/questions/41314160/java-8-hashmap-high-memory-usage/41871608#41871608) - the space could be "eaten" a lot faster. – Eugene Aug 07 '17 at 11:08

score 1 · Answer 2 · edited Aug 27 '17 at 19:15

In the Oracle Java 8 JDK, the storage for the HashMap isn't allocated until the element(s) are added.

When in doubt, just check the implementation - you can even step through it in a debugger.

Modern JDK HashMap implementations don't actually allocate the underlying array until the first element is inserted, even if you specify an explicit size. For example, on my version of the JDK 8, the constructor code is as follows:

public HashMap(int initialCapacity, float loadFactor) {
    if (initialCapacity < 0)
        throw new IllegalArgumentException("Illegal initial capacity: " +
                                           initialCapacity);
    if (initialCapacity > MAXIMUM_CAPACITY)
        initialCapacity = MAXIMUM_CAPACITY;
    if (loadFactor <= 0 || Float.isNaN(loadFactor))
        throw new IllegalArgumentException("Illegal load factor: " +
                                           loadFactor);
    this.loadFactor = loadFactor;
    this.threshold = tableSizeFor(initialCapacity);
}

Note that no array is allocated. Also, the size you have requested is larger than MAXIMUM_CAPACITY on my system which is 2³⁰, so the actual requested size (which is stored in this.threshold as described here) gets capped at MAXIMUM_CAPACITY.

Then, when you actually go to allocate the array, the implementation tries to create an array of the requested size. Ultimately, deep inside HashMap.resize() there is some logic which detects that you have reached "maximum capacity" (since you asked for an initial size of maximum capacity to start with), and sets the size of the underlying array to Integer.MAX_VALUE:

    if (newThr == 0) {
        float ft = (float)newCap * loadFactor;
        newThr = (newCap < MAXIMUM_CAPACITY && ft < (float)MAXIMUM_CAPACITY ?
                  (int)ft : Integer.MAX_VALUE);
    }

This then subsequently allocates an array of 2³¹-1 int elements, which needs at least 8G of heap space. That's why you got an OOME. When I run with -Xmx9G it successfully completes with the output:

map size: 0
map size: 1

ArrayList allocation also became lazy in j8; compare https://stackoverflow.com/questions/34250207/in-java-8-why-is-the-default-capacity-of-arraylist-now-zero — dave_thompson_085, Aug 07 '17 at 02:43

Why is this hashmap with initial capacity trying to resize?

2 Answers2