1

So I am trying to upload data larger than 16mb on a local database on an Azure Cosmos Emulator from a python app. The upload is perfect it creates the chunks along with the items. The problem I have though is on the download.

path = "eye1.zip"
fs = gridfs.GridFS(db)
with open(path, 'rb') as file1:
    grid = fs.put(file1)
    eye = fs.get(grid)
    print(eye.read())

The error I get all the time atthe .read() command is pymongo.errors.ProtocolError: Message length (12753158) is larger than server max message size (4194304). So what is the correct procedure to download the data that I have just uploaded to my device? And why is this error appearing since gridfs suppose to let you stream more than 16mb of data ?

Xristos Arthur Xenophontos
  • 21
  • 6
  • The PyMongo driver is only built and tested against the official MongoDB server. You mileage will vary against emulation environments like CosmosDB. I ran your code against MongoDB and it works. It fails on CosmosDB. I suggest you ask the folks at Microsoft for help., – Joe Drumgoole Jul 02 '20 at 15:21
  • Thank you for your answer. The problem was partially solved by making an addition in the pymongo library in file network.py in line 198 with "max_message_size=50777216". – Xristos Arthur Xenophontos Jul 07 '20 at 13:15

1 Answers1

0

Got non-interventionist solution from https://stackoverflow.com/a/56673024/4355695 : Use batch_size parameter in pymongo's .find() function:

result = mdb['collection1'].find({}, batch_size=60)

My collection had v.large records. Using batch_size=1 seemed too low; 100 was causing the same error. After some trials I settled on this 60 number as optmimum. The speed of fetching data wasn't affected. I subsequently ran the same command over thousands of such records in multiple collections and didn't get any problem.

Nikhil VJ
  • 5,630
  • 7
  • 34
  • 55