Does MongoDB index size depend on field size?

Question

In MongoDb, there is a 32-bit int type (4 byte) and a 96-bit ObjectId type (12 byte). I have noticed that index on 32-bit int field is bigger than on ObjectId field, whereas I was expecting the opposite according to this question: Are there any tools to estimate index size in MongoDB?

Is this specific to ObjectId, and how is it possible?

Here are some stats showing the difference, using MongoDB 3.2.9 and mongodb-java-driver 3.2 with default configuration ( WiredTiger engine + snappy compression level)

"_id" as ObjectId:

    > db.objectId.stats()
{
    "ns" : "test1.objectId",
    "count" : 500000,
    "size" : 20500000,
    "avgObjSize" : 41,
    "storageSize" : 6737920,
    [...]
    "nindexes" : 1,
    "totalIndexSize" : 4300800,
    "indexSizes" : {
        "_id_" : 4300800
    }
}

"_id" as int32 (linear insert):

     > db.int32linear.stats()
{
    "ns" : "test1.int32linear",
    "count" : 500000,
    "size" : 16500000,
    "avgObjSize" : 33,
    "storageSize" : 5586944,
    [...]
    "nindexes" : 1,
    "totalIndexSize" : 5255168,
    "indexSizes" : {
        "_id_" : 5255168
    }
}

"_id" as int32 (random insert):

> db.int32random.stats()
{
    "ns" : "test1.int32random",
    "count" : 500000,
    "size" : 16500000,
    "avgObjSize" : 33,
    "storageSize" : 5595136,
    [...]
    "nindexes" : 1,
    "totalIndexSize" : 5378048,
    "indexSizes" : {
        "_id_" : 5378048
    }
}

Here is the code to reproduce the test:

import com.mongodb.MongoClient;
import com.mongodb.client.MongoCollection;
import com.mongodb.client.MongoDatabase;
import java.util.ArrayList;
import java.util.Collections;
import java.util.List;
import org.bson.Document;

public class Main {

    public static void main(String[] args) {

        List<Document> listDoc = new ArrayList<>(); 

        MongoClient mongoClient = new MongoClient(); 
        MongoDatabase db = mongoClient.getDatabase("test1");

        MongoCollection<Document> objectId = db.getCollection("objectId");
        MongoCollection<Document> int32linear = db.getCollection("int32linear");
        MongoCollection<Document> int32random = db.getCollection("int32random");

        for(int i = 0; i<500000; i++){
            listDoc.add(new Document("field", "content" ));
        }
        objectId.insertMany(listDoc);
        listDoc.clear();

        for (int i = 0; i<500000; i++){
            listDoc.add(new Document("_id", i).append("field", "content"));
        }
        int32linear.insertMany(listDoc);
        // unsort the array
        Collections.shuffle(listDoc); 

        int32random.insertMany(listDoc);

        mongoClient.close();

    }
}

Which storage engine, WildTiger or mmap? Not an answer, but a couple things to try: randomized instead of ordered int values, and dropping the collection instead of deleting the documents before the test to get a clean start point as there could be freelists of deleted allocations extents if you just delete all the records. — user3973, Oct 03 '16 at 03:29
This is with wiredTiger engine. I updated the test according to your comment, but still get the same results (see my edit) — felix, Oct 03 '16 at 11:28

score 3 · Accepted Answer · answered Oct 04 '16 at 23:07

I am not certain but: WildTiger is effectively compressing the object ID keys. If you look at how they are generated, and if all the docs are inserted super quickly (in a few seconds), on a single machine, there will be a very long common prefix to the object IDs. WildTiger's key prefix compression will then be quite effective.

So why does this not works with incrementing integers? Because of the little endian format.

If the above hypothesis is correct, in practice on a real system where the insert times gap more and there are many servers (sharded) the ObjectId index would likely be a bit bigger than the int index -- but still pretty reasonable in size. If you want to check this try turning off compression on the index build.

Overall I think this is good news as the issue isn't that the int index is big, but that the ObjectId index is efficient -- ~10 bytes/entry is reasonable (although I could imagine doing better) given there is a record-id in addition to the key for each document.

https://docs.mongodb.com/manual/reference/method/ObjectId/

p.s. I believe the incrementing int index is a little smaller than the random one as there is a modest optimization in mmap for ascending keys.

you are right, if I disable prefixCompression before inserting data, index size on int32linear doesn't change, but index size on objectId double ( "indexSizes" : {"_id_" : 9818112} )! Thanks for inlighting this — felix, Oct 05 '16 at 10:51

Does MongoDB index size depend on field size?

1 Answers1