13

Quickie here that needs more domain expertise on pymongo than I have right now:

Are the "right" parts of the pymongo driver written in python for me to call gevent monkey_patch() and successfully alter pymongo's blocking behavior on r/w within gevent "asynchronous" greenlets?

If this will require a little more leg work on gevent and pymongo -- but it is feasible -- I would be more than willing to put in the time as long as i can get a little guidance over irc.

Thanks!

Note: At small scale mongo writes are not a big problem because we are just queuing a write "request" before unblocking. BUT talking to fiorix about his twisted async mongo driver (https://github.com/fiorix/mongo-async-python-driver), even mongo's quick write (requests) can cause problems in asyncronous applications at scale. (And of course, non-blocking reads could cause problems from the start!)

egbutter
  • 810
  • 11
  • 21

2 Answers2

19

I have used PyMongo with Gevent and here are a few things you need to watch out for:

  1. Instantiate only one pymongo.Connection object, preferrably as a global or module-level variable. This is important because Connection has within itself a pool!
  2. Monkey patch everything, or at least BOTH socket and threading. Due to the use of thread locals in Connection, patching socket alone is not enough.
  3. Remember to call end_request to return the connection to the pool.

The answer to your question is go ahead, PyMongo works just fine with Gevent.

Nam Nguyen
  • 1,765
  • 9
  • 13
  • thanks Bernie Hackett on mongodb google group for this update: there is still the problem that mongo has an upper limit on its connection pool much lower than the number of greenlets that most app's will be spawning if they are using mongo in the first place (1000's of greenlets). Some of the newest mongodb patches here https://github.com/mgood/mongo-python-driver are in an attempt to make gevent play nicely with mongo's thread pooling. Here is another example of a patch that *might* work: http://code.activestate.com/recipes/577490-mongodb-pool-for-gevent-and-pymongo-packages/. – egbutter Aug 26 '11 at 17:11
  • Do you have, or know of, any examples that show PyMongo being used with Gevent? – William Payne Feb 03 '12 at 01:29
  • And here is yet another example of a patch that might work: https://gist.github.com/1184264 – kkurian Feb 04 '12 at 06:36
2

On initial inspection it doesn't appear to do any socket operations in the c code so it should be fine (blocking ops should just block the green thread).

tmc
  • 716
  • 7
  • 15