0

I have a collection with ~800,000 documents, I am trying to fetch all of them, 5,000 at a time.

When running the following code:

const CHUNK_SIZE = 5000;

let skip = 0;

do {
  matches = await dbClient
    .collection(collectionName)
    .find({})
    .skip(skip)
    .limit(CHUNK_SIZE)
    .toArray();

    // ... some processing
  skip += CHUNK_SIZE;
} while (matches.length)

After about 30 iterations, I start getting documents I already received in a previous iteration.

What am I missing here?

dutzi
  • 1,880
  • 1
  • 18
  • 23

1 Answers1

0

As posted in the comments, you'll have to apply a .sort() on the query. To do so without adding too much performance overhead it would be easiest to do this on the _id e.g.

.sort(
    { 
        "_id" : 1.0
    }
)

Neither MongoDB or the AmazonDocumentDB flavor guarantees implicit result sort ordering without it.

Amazon DocumentDB

Mongo Result Ordering

Skami
  • 1,506
  • 1
  • 18
  • 29