-1

Trying to work through some training videos for asynchronous processing. Using a local Jupyter Notebook with the necessary modules installed.

Here's the code

from concurrent.futures import ProcessPoolExecutor
from concurrent.futures import as_completed
import urllib.request
import time

url_list = ['http://www.looneycorn.com/', 'http://reuters.com/', 'http://wwf.panda.org', 'https://en.unesco.org/']

def url_loader(url, time):
    with urllib.request.urlopen(url, timeout = time) as conn:
        return conn.read

def main_processpool():
    start = time.time()
    with ProcessPoolExecutor(max_workers = 1) as executor:
        future_to_page = {executor.submit(url_loader, url, 60): url for url in url_list}
        for future in as_completed(future_to_page):
            url = future_to_page[future]
            result = future.result() #this is the line that causes the error
            print('The page %r is %d bytes' % (url, len(result)))
        print('Total time taken:', time.time() - start)

main_processpool()

When I run it then the error I get is

A process in the process pool was terminated abruptly while the future was runnning or pending

I've tried several things.

  1. Trying only a single URL doesn't make it work
  2. print(future.done()) right before result = future.result() returns TRUE for all four URLs
  3. print(as_completed(future_to_page)) gives a generator object
  4. print('future =', future) shows that all four items in the for loop return have raised BrokenProcessPool. ("future =
  5. Doing everything before calling future.result() throws no errors

So how do I make this ProcessPool work?

THill3
  • 87
  • 6
  • This code results in a syntax error on line 15 (`SyntaxError: closing parenthesis ')' does not match opening parenthesis '{'`), and you have a typo in the method name. Please make sure the code is syntactically correct. – larsks Aug 07 '23 at 19:19
  • While I agree that those were there, they are hardly the cause of the issue. Nevertheless I've updated the code to fix the two typos. You'll note that the error is still thrown. – THill3 Aug 07 '23 at 23:58

0 Answers0