1

I've come across this error multiple times while using the HTTPX module. I believe I know what it means but I don't know how to solve it.

In the following example, I have an asynchronous function gather_players() that sends get requests to an API I'm using and then returns a list of all the players from a specified NBA team. Inside of teamRoster() I'm using asyncio.run() to initiate gather_players() and that's the line that produces this error: RuntimeError: The connection pool was closed while 6 HTTP requests/responses were still in-flight

async def gather_players(list_of_urlCodes):

    async def get_json(client, link):
        response = await client.get(BASE_URL + link)

        return response.json()['league']['standard']['players']

    async with httpx.AsyncClient() as client:

        tasks = []
        for code in list_of_urlCodes:
            link = f'/prod/v1/2022/teams/{code}/roster.json'
            tasks.append(asyncio.create_task(get_json(client, link)))
        
        list_of_people = await asyncio.gather(*tasks)
        
        return list_of_people

def teamRoster(list_of_urlCodes: list) -> list:
        list_of_personIds = asyncio.run(gather_players(list_of_urlCodes))

        finalResult = []
        for person in list_of_personIds:
            personId = person['personId']

            #listOfPLayers is a list of every NBA player that I got 
            #from a previous get request
            for player in listOfPlayers:
                if personId == player['personId']:
                    finalResult.append({
                        "playerName": f"{player['firstName']} {player['lastName']}",
                        "personId": player['personId'],
                        "jersey": player['jersey'],
                        "pos": player['pos'],
                        "heightMeters": player['heightMeters'],
                        "weightKilograms": player['weightKilograms'],
                        "dateOfBirthUTC": player['dateOfBirthUTC'],
                        "nbaDebutYear": player['nbaDebutYear'],
                        "country": player['country']
                    })

        return finalResult

*Note: The teamRoster() function in my original script is actually a class method and I've also used the same technique with the asynchronous function to send multiple get request in an earlier part of my script.

Tony
  • 266
  • 1
  • 11

2 Answers2

5

I was able to finally find a solution to this problem. For some reason the context manager: async with httpx.AsyncClient() as client fails to properly close the AsyncClient. A quick fix to this problem is closing it manually using: client.aclose()

Before:

async with httpx.AsyncClient() as client:

    tasks = []
    for code in list_of_urlCodes:
        link = f'/prod/v1/2022/teams/{code}/roster.json'
        tasks.append(asyncio.create_task(get_json(client, link)))

    list_of_people = await asyncio.gather(*tasks)

    return list_of_people

After:

client = httpx.AsyncClient()

tasks = []
for code in list_of_urlCodes:
    link = f'/prod/v1/2022/teams/{code}/roster.json'
    tasks.append(asyncio.create_task(get_json(client, link)))

list_of_people = await asyncio.gather(*tasks)
client.aclose()    

return list_of_people
Tony
  • 266
  • 1
  • 11
1

The accepted answer claims that the original code failed to properly close the client because it didn't call aclose(), and while that's technically true the implementation of the async context manager exit method (__aexit__) essentially duplicates the aclose() implementation.

In fact, you can tell that the connection is closed because the error message complains about 6 HTTP requests remaining in-flight after the connection is closed.

By contrast, the accepted answer "fixes" the error by explicitly not closing the connection. Because httpx.AsyncClient.aclose is an async function, calling it without awaiting creates a coroutine that is not actually scheduled for execution on the event loop. That coroutine is then destroyed when the function returns immediately after without having ever actually executed, meaning the connection is never closed. Python should print a RuntimeWarning that client.aclose() was never awaited. As a result, each request has plenty of time to complete before the process terminates and force-closes each connection so the RuntimeError is never raised.

While I don't know the full reason that some requests were still in-flight, I suspect it was some cleanup at the end that didn't finish before the function returned and the connections were closed. For instance, if you put await asyncio.sleep(1) right before the return, then the error would likely go away as the client would have time to finish and clean up after each of its requests. (Note I'm not saying this is a good fix, but rather would help provide evidence to back up my explanation.)

Instead of using asyncio.gather, try using TaskGroups as recommended by the Python docs for asyncio.gather. So your new code could look something like this:

async def gather_players(list_of_urlCodes):

    async def get_json(client, link):
        response = await client.get(BASE_URL + link)

        return response.json()['league']['standard']['players']

    async with httpx.AsyncClient() as client:

        async with asyncio.TaskGroup() as tg:
            tasks = [tg.create_task(get_json(client, f'/prod/v1/2022/teams/{code}/roster.json')) for code in list_of_urlCodes]

        list_of_people = [task.result for task in tasks]
        
    return list_of_people

This is obviously not production-grade code, as it is missing error-handling, but demonstrates the suggestion clearly enough.

Jason Swails
  • 31
  • 1