0

I have a computationally heavy project that appears to be a good candidate for parallelization. The code uses a large number of GIS contours and takes up roughly 1.5 GB of memory as a single process. There are two levels of logic that can be parallelized: an outer level that splits the project area into smaller (but still rather larger) areas, and an inner loop that does a lot of math with pretty short segements.

Attempts to use concurrent.futures on the outer loop failed due to pickle. Pathos ran and created multiple process but used a ton of memory and was actually slower. I'm assuming the slow down was due to dill serializing and recreating very large objects.

I haven't attempt to parrallelize the inner loop yet but should be able to break the code and contours into relatively small objects (~10 KB), not counting external modules (shapely). While there are a plethora of parallel processing options for python I haven't found good discussion on the best way handle objects and managing memory. What is a good package/method to efficiently split off a small object into a new process from a much larger process?

I think it would be preferable to use both levels of parrallelization, though I'm not sure and need to do some profiling. The outer loop can be made slightly more memory efficient but it may not be realistic to split it into multiple processes. The inner loops should be easy to break into small, memory efficient pieces, I'm just not sure of the best package to use.

Edit: The vast majority of the processing is within the shapley package which is a python front end for GEOS which is in C (I think). I'm currently using python 2.7 but am willing to switch to 3 if it is beneficial.

Mike Bannister
  • 129
  • 2
  • 8
  • 1
    Python's built-in `multiprocessing` module has always worked pretty well for me. It may be worth looking into doing the computationally heavy steps in C or C++ though as they will be a good bit quicker than Python unless you're already using Python libraries that use a compiled language internally. – Daniel Underwood May 26 '16 at 17:42
  • Yes, what @danielu13 said is probably the "best" way of doing it properly. I use `threading` sometimes when I need to speed up small, repetetive tasks just because it is easy to use. – Pax Vobiscum May 26 '16 at 17:42
  • @Uzzee regarding the `threading` module -- Does it not just provide a way to do multiple things at once rather than using multiple OS threads for true concurrency? That's why I have always used `multiprocessing` since it uses multiple Python processes and thus uses multiple cores. – Daniel Underwood May 26 '16 at 18:12
  • @danielu13 well, https://wiki.python.org/moin/GlobalInterpreterLock – Pax Vobiscum May 26 '16 at 20:34
  • I'm the `dill`/`pathos` author. I think I'd agree about the cause being `dill` eating memory due to some large objects. You might get some better mileage by using one of the alternate `dill.settings`. However, "memory-efficient" would be a good setting to add… Hmm. – Mike McKerns May 29 '16 at 03:25

0 Answers0