Moving records from one collection to another PyMongo

Question

What is the proper way of moving a number of records from one collection to another. I have come across several other SO posts such as this which deal with achieve the same goal but none have a python implementation.

#Taking a number of records from one database and returning the cursor
cursor_excess_new = db.test_collection_new.find().sort([("_id", 1)]).limit(excess_num)


# db.test.insert_many(doc for doc in cursor_excess_new).inserted_ids


# Iterating cursor and Trying to write to another database
# for doc in cursor_excess_new:
#     db.test_collection_old.insert_one(doc)
result = db.test_collection_old.bulk_write([
    for doc in cursor_excess_new:
        InsertMany(doc for each doc in cur)
        pprint(doc)
])

If I use insert_many, I get the following error: pymongo.errors.OperationFailure: Writes to config servers must have batch size of 1, found 10

bulk_write is giving me a syntax error at the start of for loop.

What is the best practice and correct way of transferring records from one collection to another in pymongo so that it is atomic?

score 1 · Accepted Answer · answered Dec 05 '17 at 04:20

Collection.bulk_write accepts as argument an iterable of query operations.

pymongo has pymongo.operations.InsertOne operation not ~~InsertMany~~.

For your situation, you can build a list of InsertOne operations for each document in the source collection. Then do a bulk_write on the destination using the built-up list of operations.

from pymongo import InsertOne
...
cursor_excess_new = (
    db.test_collection_new
      .find()
      .sort([("_id", 1)])
      .limit(excess_num)
)

queries = [InsertOne(doc) for doc in cursor_excess_new]
db.test_collection_old.bulk_write(queries)

score 1 · Answer 2 · answered Aug 11 '19 at 11:37

You don't need "for loop".

   myList=list(collection1.find({}))
   collection2.insert_many(myList)
   collection1.delete_many({})

If you need to filter it you can use the following code:

   myList=list(collection1.find({'status':10}))
   collection2.insert_many(myList)
   collection1.delete_many({'status':10})

But be careful, because it has no warranty to move successfully so you need to control transactions. If you're going to use the following code you should consider that your MongoDb shouldn't be Standalone and you need to active Replication and have another instance.

  with myClient.start_session() as mySession:
      with mySession.start_transaction():
          ...yourcode...

Finally, the above code has the warranty to move (insert and delete) successfully but the transaction isn't in your hand and you can't get the result of this transaction so you can use the following code to control both moving and transaction:

  with myClient.start_session() as mySession:
      mySession.start_transaction()
      try:
          ...yourcode...
          mySession.commit_transaction()
          print("Done")
      except Exception as e:
          mySession.abort_transaction()
          print("Failed",e)

Moving records from one collection to another PyMongo

2 Answers2