0

I am load testing my system using the Locust framework. My project contains the Locust source files (not listed as dependency) because I had to make some modifications to the source code for some needed functionality. Unfortunately in locust/core the following code is called

from gevent import GreenletExit, monkey
from six.moves import xrange
monkey.patch_all()

In a different part of my project (outside of locust - but still dependent upon the locally modified version of locust) I am trying to use the multiprocessing module:

from multiprocessing import Pool

def dummy_func
pool = Pool(4) #gets hung up here -- likly because of patched sockets in os module
results = pool.map(irrelevant_func, irrelevant_data)

The issue is the monkey.patch_all() patches the multiprocessing modules dependencies (such as os module) making it unusable. I can't remove the monkey.patch_all() call as its essential to Locust, but I need to access the original version of the multiprocessing module (and all its dependencies) so I can retain its functionality.

TLDR: Gevent's MonkeyPatching causes Multiprocessing Pool to hang (wont initialize) because it patches the os modules socket -- how do I get a version of Multiprocessing that accesses an unpatched os module along with a version of gevent that accesses a patched os module

My Research for similar bug

  1. Python unit test best practice to avoid monkey patching bug
  2. Gevent monkeypatching breaking multiprocessing
  3. https://groups.google.com/forum/#!topic/gevent/Mjk3qc0i13E
spencer.pinegar
  • 442
  • 2
  • 10
  • Looking at the [gevent monkey.py source](https://github.com/gevent/gevent/blob/master/src/gevent/monkey.py), I don't see `gevent` patching the `multiprocessing` module at all. – zwer Jul 24 '18 at 23:51
  • @zwer Unfortunately it patches modules it is dependent on for concurrency..... such as the os directory – spencer.pinegar Jul 25 '18 at 15:20
  • Have you tried accessing the `saved` dictionary from `gevent.monkey`, i.e. `multiprocessing = gevent.monkey.saved["multiprocessing"]`? The problem might be a bit deeper than that, tho, you might need to trace through the patches and create your own parallel version referencing pristine modules. – zwer Jul 25 '18 at 16:01

0 Answers0