0

I tested various proxies to query a public api using the Python module requests. My requests came through fine.

Unfortunately using aiohttp with asyncio, I get the following error:

ClientHttpProxyError: 403, message='Forbidden', url=URL('http://165.225.56.117:10605')

My code looks like so:

def fire_requests(urls, proxy):
    results = []
    async def gather_with_concurrency(n, proxy):
        semaphore = asyncio.Semaphore(n)
        session = aiohttp.ClientSession(connector=conn, trust_env=True)

        async def get(url, proxy):
            async with semaphore:
                async with session.get(url, ssl=False, proxy=proxy) as response:
                    obj = json.loads(await response.read())
                    results.append(obj)
        await asyncio.gather(*(get(url, proxy) for url in urls))
        await session.close()
    # Initialize connection pool
    conn = aiohttp.TCPConnector(limit_per_host=100, limit=0, ttl_dns_cache=300)
    PARALLEL_REQUESTS = 240
    loop = asyncio.get_event_loop()
    loop.run_until_complete(gather_with_concurrency(PARALLEL_REQUESTS, proxy))
    request_finished = datetime.now()
    conn.close()
    return request_finished, results
  • Perhaps authentication? [like this](https://stackoverflow.com/a/53160989/6242321) – jwal Oct 01 '22 at 23:07
  • Doesn't make too much sense if it works w/o when using a standard requests call, as I described no? – Google Google Oct 02 '22 at 11:25
  • You should read [MRE](https://stackoverflow.com/help/minimal-reproducible-example). My *guess* is that you are assuming identical behaviour between requests and aiohttp that you have not provided in your question. Environment variables on proxies are one such difference. – jwal Oct 02 '22 at 16:53
  • My answer is that I am not using a proxy that requires auth, so it does not make sense to argue about auth. – Google Google Oct 04 '22 at 07:25
  • Have a look at the MRE link, then I'd not have to guess. – jwal Oct 04 '22 at 10:33

0 Answers0