5

I am reading the Multiprocessing topic for Python 3 and trying to incorporate the method into my script, however I receive the following error:

AttributeError: __ exit __

I use Windows 7 with an i-7 8-core processor, I have a large shapefile which I want processed (with the mapping software, QGIS) using all 8 cores preferably. Below is the code I have, I would greatly appreciate any help with this matter:

from multiprocessing import Process, Pool

def f():
    general.runalg("qgis:dissolve", Input, False, 'LAYER_ID', Output)

if __name__ == '__main__':
    with Pool(processes=8) as pool:
        result = pool.apply_async(f)
Joseph
  • 586
  • 1
  • 13
  • 32
  • 1
    First thing I would do is replace the content of f() with, for example print to narrow down the issue. If that passes then there is something wrong with your general.runalg – Tymoteusz Paul Apr 27 '15 at 11:36
  • @TymoteuszPaul - Thank you, I will try this. The `general.runal` works fine on its own so my guess is something to do with the `Pool` method. – Joseph Apr 27 '15 at 11:37
  • Also you've mentioned that you are writing this for python3, and yet you are using print in 2.7 fashion which will throw an error. And may be the cause of all things ;) – Tymoteusz Paul Apr 27 '15 at 11:38
  • @TymoteuszPaul - I wish :). Removed `print` but still get an error so will continue your first suggestion. – Joseph Apr 27 '15 at 11:42
  • 1
    Is `dissolve` defined somewhere before passing it to apply_async? – chown Apr 27 '15 at 12:58
  • @chown - Thanks buddy, my apologies but no, `dissolve` is not defined anywhere else (my mistake). I've edited the question. – Joseph Apr 27 '15 at 13:06

1 Answers1

3

The context manager feature of multiprocessing.Pool was only added into Python 3.3:

New in version 3.3: Pool objects now support the context management protocol – see Context Manager Types. __enter__() returns the pool object, and __exit__() calls terminate().

The fact that __exit__ is not defined suggests you're using 3.2 or earlier. You'll need to manually call terminate on the Pool to get equivalent behavior:

if __name__ == '__main__':
    pool = Pool(processes=8)
    try:
        result = pool.apply_async(f)
    finally:
        pool.terminate()

That said, you probably don't want to use terminate (or the with statement, by extension) here. The __exit__ method of the Pool calls terminate, which will forcibly exit your workers, even if they're not done with their work. You probably want to actually wait for the worker to finish before you exit, which means you should call close() instead, and then use join to wait for all the workers to finish before exiting:

if __name__ == '__main__':
    pool = Pool(processes=8)
    result = pool.apply_async(f)
    pool.close()
    pool.join()
dano
  • 91,354
  • 19
  • 222
  • 219
  • Brilliant, many thanks for this! I can see the 8 separate processes ocurring in the Task Manager. The function doesn't run correctly but I think that's in my end so will play around with the script. Thanks again! – Joseph Apr 27 '15 at 15:13