Trying to work through some training videos for asynchronous processing. Using a local Jupyter Notebook with the necessary modules installed.
Here's the code
from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import as_completed
import urllib.request
import time
url_list = ['http://www.looneycorn.com/', 'http://reuters.com/', 'http://wwf.panda.org', 'https://en.unesco.org/']
def url_loader(url, time):
with urllib.request.urlopen(url, timeout = time) as conn:
return conn.read
def main_processpool():
start = time.time()
with ProcessPoolExecutor(max_workers = 1) as executor:
future_to_page = {executor.submit(url_loader, url, 60): url for url in url_list}
for future in as_completed(future_to_page):
url = future_to_page[future]
result = future.result() #this is the line that causes the error
print('The page %r is %d bytes' % (url, len(result)))
print('Total time taken:', time.time() - start)
main_processpool()
When I run it then the error I get is
A process in the process pool was terminated abruptly while the future was runnning or pending
I've tried several things.
- Trying only a single URL doesn't make it work
print(future.done())
right beforeresult = future.result()
returns TRUE for all four URLsprint(as_completed(future_to_page))
gives a generator objectprint('future =', future)
shows that all four items in the for loop return haveraised BrokenProcessPool
. ("future =- Doing everything before calling
future.result()
throws no errors
So how do I make this ProcessPool work?