Mongodb - updating a collection with >100,000 documents in migration

Question

Following migration code seems to be getting stuck while updating this huge collection.

@ChangeSet(order = "010", id = "add_tenant_to_product", author = "pn")
fun addTenantToProduct(mongoTemplate: MongoTemplate) {
    log.info("Adding tenants to products")
    val query = Query()
    val update = Update()
    update.set("tenants", arrayOf("shared"))

    mongoTemplate.updateMulti(query, update, Product::class.java)
}

@ChangeSet(order = "011", id = "add_tenant_to_product_version", author = "pn")
fun addTenantToProductVersion(mongoTemplate: MongoTemplate) {
    log.info("Adding tenants to product-version")
    val query = Query()
    val update = Update()
    ...

The last few lines of the application log indicate the changeset getting invoked but subsequent one not getting invoked.

2021-05-11 08:00:29.385 INFO [product-service,,,] 1 --- [
main] c.a.a.mongock.changelog.Migration : Adding tenants to products

2021-05-11 08:00:29.576 INFO [product-service,,,] 1 --- [
main] org.mongodb.driver.connection : Opened connection

The Spring actuator health probe fails because of this and the container keeps on restarting. Is there a better API to use which will be faster?

Note:

1. version used: 'com.github.cloudyrock:mongock:1.12'
2. below is the configuration

@Profile("!tenant")
@Configuration
class MongockConfig : AbstractMongoConfiguration() {

    @Value("\${spring.data.mongodb.uri}")
    private lateinit var uri: String

    @Value("\${spring.data.mongodb.database}")
    private lateinit var database: String

    override fun mongoClient(): MongoClient {
        return MongoClient(MongoClientURI(uri))
    }

    override fun getDatabaseName(): String {
        return database
    }

    @Bean
    fun mongock(): Mongock {
        return MongockBuilder(mongoClient(), databaseName, "com.somepackage.mongock.changelog")
            .setMongoTemplate(mongoTemplate())
            .setLockQuickConfig()
            .build()
    }
}

Here are the logs

---

2021-05-10 10:59:33.406 INFO [product-service,,,] 1 --- [ main] org.mongodb.driver.connection : Opened connection [connectionId{localValue:16, serverValue:26359638}] to dummy-clusterr-39649.servers.mongodirector.com:27017 2021-05-10 10:59:33.525 INFO [product-service,,,] 1 --- [ main] c.github.cloudyrock.mongock.LockChecker : Mongbee trying to acquire the lock 2021-05-10 10:59:33.574 INFO [product-service,,,] 1 --- [ main] c.github.cloudyrock.mongock.LockChecker : Lock is taken by other process until: Mon May 10 11:01:03 GMT 2021 2021-05-10 10:59:33.574 INFO [product-service,,,] 1 --- [ main] c.github.cloudyrock.mongock.LockChecker : Mongock is going to sleep to wait for the lock: 90265 ms(1 minutes) 2021-05-10 11:01:03.839 INFO [product-service,,,] 1 --- [ main] c.github.cloudyrock.mongock.LockChecker : Mongbee trying to acquire the lock 2021-05-10 11:01:03.844 INFO [product-service,,,] 1 --- [ main] c.github.cloudyrock.mongock.LockChecker : Mongbee acquired the lock until: Mon May 10 11:04:03 GMT 2021 2021-05-10 11:01:03.844 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : Mongock starting the data migration sequence.. 2021-05-10 11:01:03.982 INFO [product-service,,,] 1 --- [ main] org.reflections.Reflections : Reflections took 60 ms to scan 1 urls, producing 2 keys and 3 values 2021-05-10 11:01:04.038 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@7abd0e71 pass over 2021-05-10 11:01:04.046 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@6635f36 pass over 2021-05-10 11:01:04.047 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@61352a80 pass over 2021-05-10 11:01:04.052 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@7576ed14 pass over 2021-05-10 11:01:04.054 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@b0302179 pass over 2021-05-10 11:01:04.068 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@e6caade0 pass over 2021-05-10 11:01:04.071 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@2688129a pass over 2021-05-10 11:01:04.073 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@e46af677 pass over 2021-05-10 11:01:04.075 INFO [product-service,,,] 1 --- [ main] com.github.cloudyrock.mongock.Mongock : com.github.cloudyrock.mongock.ChangeEntry@d788b92f pass over 2021-05-10 11:01:04.077 INFO [product-service,,,] 1 --- [ main] c.a.a.mongock.changelog.Migration : Adding organisations to questions 2021-05-10 11:01:04.158 INFO [product-service,,,] 1 --- [ main] org.mongodb.driver.connection : Opened connection [connectionId{localValue:17, serverValue:26359725}] to dummy-clusterr-39649.servers.mongodirector.com:27017

Answer 1

can you provide a bit more information, such as mongock version you are using, log trace(what you show is not enough fo diagnostic), mongock configuration, etc?

On the other hand, you should use MongockTempalte instead MongoTemplate directly. It's the same API, you wont miss anything. Actually using a MongoTemplate directly should throw an exception.

For more information, please check Mongock's documenation