Pymongo Aggregation Pipeline is returning only 100 Documents

Question

I have a simple pipeline and running fine in MongoDB Compass, but the same pipeline when I try to execute using Pymongo I got less number of documents. When I checked the result cursor is having or limiting only the 100 documents, but in Compass I see more data returned. I am using the below connection but not sure what I am missing.

The expected documents less than 200 documents.

Python version: 3.7
Pymongo version: pymongo==3.11.3


db.collections.aggregate([
    {
        '$match': {
            'subject': {
                '$in': [
                    '1','2','3'
                ]
            }
        }
    }, {
        '$lookup': {
            'from': 'schools', 
            'localField': 'locId', 
            'foreignField': 'locId', 
            'as': 'schools'
        }
    }, {
        '$unwind': {
            'path': '$schools', 
            'preserveNullAndEmptyArrays': True
        }
    }, {
        '$unwind': {
            'path': '$schools.schools', 
            'preserveNullAndEmptyArrays': False
        }
    }, {
        '$match': {
            'schools.appointmentDate': {
                '$gte': datetime(2021, 3, 15, 0, 0, 0, tzinfo=timezone.utc), 
                '$lt': datetime(2021, 3, 16, 0, 0, 0, tzinfo=timezone.utc)
            }, 
            'schools.schools.appointmentStatus': {
                '$in': [
                    'Scheduled'
                ]
            }
        }
    }, {
        '$group': {
            '_id': '$subject', 
            'totalAppointments': {
                '$sum': 1
            }
        }
    }
])

I have also used {"$limit" : 10000} after the sort as well in the mongo query, but not helping me.

Can some one please help me what I have to check?

Thanks

Reduce your code to only required parts. Remove str(document), allow disk use, specify a trivial pipeline etc. — D. SM, Mar 02 '21 at 23:42

score 0 · Accepted Answer · answered Mar 09 '21 at 03:55

I am not sure why "aggregate" was not returning the complete result set for me, So I tried this alternate approach using "aggregate_raw_batches" and resolved my issues.

import bson
cursor = db.test.aggregate_raw_batches(pipeline) 
for batch in cursor:
    for i in bson.decode_all(batch):
         #do with documents..

Pymongo Aggregation Pipeline is returning only 100 Documents

1 Answers1