3

I have a twisted server application that maintains persistent connections with around 1000 clients. Each client will occasionally send data to my twisted server, and I would like that server to store data into a MongoDB database. So far, so good.

But, the pymongo documentation states something like "there is no good way to use pymongo with twisted". I don't understand why not. Can someone please explain what exactly the problem is and what the pitfalls are? I think it has something to do with pymongo being synchronous, but all I want to do is shove some stuff into the database.

If I have a single instance of pymongo.MongoClient declared in a file connector.py, and then I import connector into the main python file that runs my Twisted factory and protocols, I should be able to use that instance of pymongo.MongoClient in each Protocol (connection to each client) to store the data to the database.

So what exactly is the problem? Obviously I'm a bit confused.

Marc
  • 3,386
  • 8
  • 44
  • 68

1 Answers1

5

In short: pymongo calls are blocking, when ran they freeze the twisted engine until the call returns, which is randomly destructive to twisted internal state because it's the opposite of what twisted is designed for.

Instead you should look for a twisted comparable driver such as tx-mongo. tx-mongo has a smaller audience then pymongo, and so its documentation is comparably a bit rough but you should be able to find all you need in it's example directory.

Background: Threading vs Event programing

Community
  • 1
  • 1
Mike Lutz
  • 1,812
  • 1
  • 10
  • 17
  • Ok thanks. But all I am doing is just inserting a document in a collection using the collection.insert function. I don't need to see the result of this call - it is just fire and forget. Suppose I wrap this call in a thread from the python threading library. Will this still cause problems? – Marc Aug 28 '14 at 22:32
  • Threading is unlikely to help unless your running something other then CPython because of the [GIL](http://en.wikipedia.org/wiki/Global_Interpreter_Lock). If your app has tons of headroom you might be able to get away with doing the pymongo call, its just that it will fail very unpredictably and ungracefully (and be a bear to debug if you forgot you cheated this way). https://github.com/fiorix/mongo-async-python-driver is the twisted native mongo driver I've had good luck with in the past, give that a look before you do something you'll regret later. – Mike Lutz Aug 28 '14 at 23:06
  • Threading may help because the GIL is released around blocking I/O operations implemented in the CPython standard library. Essentially, you can get as much thread-based concurrency by using blocking APIs with multithreading if you *are* using Twisted as if you *aren't* using Twisted (in other words, Twisted doesn't make threads worse, it just doesn't make them better). – Jean-Paul Calderone Aug 29 '14 at 01:18