MongoDB stuck creating index with "Index Build: draining writes received during build" message

Question

I have 4 rows in the test collection:

{ "_id" : ObjectId("5f4ce50e19b13337216dd477"), "test" : 1 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd478"), "test" : 2 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd479"), "test" : 3 }
{ "_id" : ObjectId("5f4ce50e19b13337216dd47a"), "test" : 4 }

After running db.test.createIndex({test:1},{background:1}); to create an index, it just hangs. It was hanging for at least a few hours. Here is what I found in the db.currentOp() about this operation:

{
    "type" : "op",
    "host" : "HOSTNAME:27017",
    "desc" : "IndexBuildsCoordinatorMongod-13",
    "active" : true,
    "currentOpTime" : "2020-08-31T12:11:13.159+00:00",
    "opid" : 8721867,
    "secs_running" : NumberLong(20),
    "microsecs_running" : NumberLong(20888590),
    "op" : "command",
    "ns" : "test.test",
    "command" : {
        "createIndexes" : "test",
        "indexes" : [
            {
                "v" : 2,
                "key" : {
                    "test" : 1
                },
                "name" : "test_1",
                "background" : 1
            }
        ],
        "lsid" : {
            "id" : UUID("07b43083-8ab9-4bcb-8768-919a3f27655f")
        },
        "$clusterTime" : {
            "clusterTime" : Timestamp(1598875647, 409),
            "signature" : {
                "hash" : BinData(0,"+/YcdPyQriT8RL1LtFUhxe2BtCE="),
                "keyId" : NumberLong("6861636045532823556")
            }
        },
        "$db" : "test"
    },
    "msg" : "Index Build: draining writes received during build",
    "numYields" : 0,
    "locks" : {

    },
    "waitingForLock" : false,
    "lockStats" : {
        "ReplicationStateTransition" : {
            "acquireCount" : {
                "w" : NumberLong(6)
            }
        },
        "Global" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(4)
            }
        },
        "Database" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(4)
            }
        },
        "Collection" : {
            "acquireCount" : {
                "r" : NumberLong(1),
                "w" : NumberLong(3),
                "W" : NumberLong(1)
            }
        },
        "Mutex" : {
            "acquireCount" : {
                "r" : NumberLong(4)
            }
        }
    },
    "waitingForFlowControl" : false,
    "flowControlStats" : {
        "acquireCount" : NumberLong(3),
        "timeAcquiringMicros" : NumberLong(1)
    }
}

This Index Build: draining writes received during build makes no sense since there was no read/writes to the test collection during index creation.

Also index creation hangs only in non-empty collection. Index creates successfully in empty collection.

What might be an issue in this case? I'm out of ideas.

I would report this using the official support channels. – D. SM Aug 31 '20 at 16:21 — D. SM, Aug 31 '20 at 16:21

score 3 · Accepted Answer · answered Sep 03 '20 at 11:39

Finally figured it out with the help of MongoDB team. The node can't communicate with itself so it will hang trying to commit the index build. This was the reason. Adding keyfile fixed the issue:

rm -f mongo.keyfile
openssl rand -base64 756 > mongo.keyfile
chmod 400 mongo.keyfile
 
bin/mongod --config mongo.conf --keyFile mongo.keyfile

Here are the links to the MongoDB's Jira issues which cover this subject: https://jira.mongodb.org/browse/SERVER-50665 and https://jira.mongodb.org/browse/SERVER-48516

some of the replicas being down may cause this issue as well. — Ali, Apr 30 '21 at 23:45

score 0 · Answer 2 · answered Mar 17 '23 at 18:29

To authenticate with MongoDB using a keyfile, you need to follow these steps:

Create a keyfile using the openssl command line tool. For example:

openssl rand -base64 741 > /path/to/keyfile

This command will create a keyfile with 741 random bytes and store it at /path/to/keyfile.

Make sure that the keyfile is only readable by the user running the mongod process. For example:

chmod 400 /path/to/keyfile

This command will set the file permissions to read-only for the file owner.

Start the mongod process with the --keyFile option. For example:

mongod --auth --keyFile /path/to/keyfile

This command will start the mongod process with authentication enabled and using the specified keyfile for authentication.

To authenticate with the MongoDB server using the keyfile, use the --authenticationMechanism option with the value MONGODB-X509. For example:

mongo --host myhost.example.com --authenticationMechanism MONGODB-X509 --ssl --sslCAFile /path/to/ca.pem --sslPEMKeyFile /path/to/client.pem

This command will authenticate with the MongoDB server at myhost.example.com using the X.509 authentication mechanism and SSL encryption. The --sslCAFile option specifies the path to the CA certificate, and the --sslPEMKeyFile option specifies the path to the client certificate containing the public key and private key signed by the CA certificate.

Note that X.509 authentication is based on certificates rather than passwords, so you don't need to specify a username or password.

MongoDB stuck creating index with "Index Build: draining writes received during build" message

2 Answers2