How do I get the body of an HTTP 403 response using Python+aiohttp?

Question

I'm using Python 3.6 and the aiohttp library to make an API Post request to a server. If I use the wrong username when making the request, I get an HTTP 403 error as I expect. When I make this request in Postman, the body of the response shows:

{"error_message": "No entitlements for User123"}

When I make the request using aiohttp however, I don't see this response body anywhere. The message just says "Forbidden". How can I get the error message above in my Python code?

Edit: Here's my aiohttp code, although it's pretty straightforward:

try:
    async with self.client_session.post(url, json=my_data, headers=my_headers) as response:
        return await response.json()
except ClientResponseError as e:
    print(e.message)  # I want to access the response body here
    raise e

Edit 2: I found a workaround. When I'm creating the client_session, I set the raise_for_status value to False. Then when I get a response from the API call, I check if the status is >= 400. If so, I handle the error myself, which includes the body of the response.

Edit 3: Here's the code for my workaround:

self.client_session = ClientSession(loop=asyncio.get_event_loop(), raise_for_status=False)
####################### turn off the default exception handling ---^

try:
    async with self.client_session.post(url, json=my_data, headers=my_headers) as response:
    body = await response.text()

    # handle the error myself so that I have access to the response text
    if response.status >= 400:
        print('Error is %s' % body)
        self.handle_error(response)

Try to read response as bytes or as text instead of json (`await response.read()` or `await response.text()`). May be that problem in content type of response or in other place. — Yurii Kramarenko, Mar 13 '19 at 15:53
@YuriiKramarenko control never reaches that `return await response.json()` line. It skips right to the `except` block — user2023861, Mar 13 '19 at 16:01
@user2023861 Show your `ClientSession` initialization please — Yurii Kramarenko, Mar 13 '19 at 16:04
@user2023861 You should use `raise_for_status()` that basically raises and error if the status code is >= 400. The link from official docs: https://docs.aiohttp.org/en/stable/client_reference.html#aiohttp.ClientResponse.raise_for_status — Bogdan Condurache, Mar 13 '19 at 16:08
@user2023861 also workaround, that you find, is correct. Just remove `raise_for_status` from your `ClientSession` initialization and process error in `RequestContextManager` — Yurii Kramarenko, Mar 13 '19 at 16:12
@YuriiKramarenko I edited my question to include my workaround code. I don't know enough about the RequestContextManager to change it, but my workaround is pretty simple — user2023861, Mar 13 '19 at 19:28
@BogdanCondurache that raise_for_status() function raises a ClientResponseError that doesn't include the response body. I need access to the response body. Hence my question — user2023861, Mar 13 '19 at 19:29
@user2023861 `async with self.client_session.post(url, json=my_data, headers=my_headers) as response` returns you `RequestContextManager`. You can't access response body outside of it. So just use workaround that you already found — Yurii Kramarenko, Mar 14 '19 at 12:16

Zaar Hai · Answer 1 · 2020-12-14T07:08:12.010

Yes, it may be indeed confusing if you are coming from requests package that has its exception objects having .request.response (or the other way around) attribute.

You've already figured that out obviously, but here is a proper answer for the older versions of aiohttp.

async with session.post(...) as response:
    try:
        response.raise_for_status()
    except ClientResponseError as err:
        logger.error("Error: %s, Error body: %s", err, (await response.text()))

    return await response.json()

The newer versions unfortunately recycle the connection once raise_for_status() is called hence you can't fetch the error body later. Here is what work for me nowadays (from http-noah package):

        logger = structlog.get_logger(__name__)

        async with session.post(url, **req_kwargs) as res:
            # Fetching text as error just in case - raise_for_status() will
            # release the connection so the error body will be lost already.
            # The text will be cached in the response internally for the use later on
            # so no waste here.
            err_body = await res.text()
            try:
                res.raise_for_status()
            except aiohttp.ClientResponseError as err:
                logger.error("Request failed", err=err, err_body=err_body)
                raise

This code still gives me a "Connection closed" exception when trying to read the response text in the exception handler (aiohttp 3.7.3, python 3.6) File ".../site-packages/aiohttp/streams.py", line 344, in read raise self._exception aiohttp.client_exceptions.ClientConnectionError: Connection closed — John, Dec 10 '20 at 16:11
You are right - in aiohttp 3.7.x (I think) they've "fixed" it by causing `raise_for_status()` to recycle the connection back into pool. Here is what works: https://github.com/haizaar/http-noah/blob/master/http_noah/async_client.py#L203 — Zaar Hai, Dec 14 '20 at 07:03

How do I get the body of an HTTP 403 response using Python+aiohttp?

1 Answers1