Get aiohttp request response in callback with a infinite loop of asyncio task

Question

What I am doing- I am trying to make multiple post requests(to the same URL with different parameters) in an infinite loop using asyncio and aiohttp.

def callback(response):
    print(response) 

async def make_request(session, payload, sema):
        async with sema, session.post(url=URL, headers=HEADERS, data=json.dumps(payload)) \
                as response:
            return await response.json()


async def generate_requests(payloads):
    tasks = []
    sema = asyncio.BoundedSemaphore(value=100)
    async with aiohttp.ClientSession() as session:
        for payload in payloads:
            tasks.append(make_request(session, payload, sema))
        results = await asyncio.gather(*tasks, return_exceptions=True)
        print(results)


def run_asyncio_requests():
    loop = asyncio.get_event_loop()
    while True:
        for country in countries:
            cities = country.cities
            loop.run_until_complete(generate_requests(cities))


if __name__ == '__main__':
    run_asyncio_requests()

Problem: run_asyncio_requests function generates and make multiple requests(cities against a country) and then await for all responses for one country requests then executes the next country requests.

What I want: All countries' requests should be executed at once and callback(some kind of linked function with each request) function should handle each response because post request response time for each request is different. It should not wait to gather all the responses. Requests should be generated in each cycle(infinite-loop) and callback function in the background should be linked with each response. I know I am using asyncio.gather to get all responses until requests completed. But I have no idea how to do that.

score 0 · Answer 1 · answered Mar 03 '20 at 14:42

0

You can apply the same technique when invoking generate_requests that you did when invoking make_request:

async def generate_all_countries():
    tasks = [generate_requests(country.cities) for country in countries]
    results = await asyncio.gather(*tasks)
    print(results)

if __name__ == '__main__':
    asyncio.run(generate_all_countries())

Note that you don't need run_asyncio_requests as a separate sync function, you can simply invoke an async function using asyncio.run() and proceed from there.

answered Mar 03 '20 at 14:42

user4815162342

141,790
18
296
355

Thanks for your response. Yes, your suggested code will generate all requests what about the callback function that will collect response when a response comes. – Ahsan aslam Mar 04 '20 at 09:24
@Ahsanaslam Can't you just modify `make_request` and/or `generate_requests` to invoke the callback function you need? – user4815162342 Mar 04 '20 at 09:59
I have no idea where to place callback function and one more thing is that In one cycle(infinite-loop) all countries request generated and await for all response but I need to continue infinite-loop that generates continuously and callback function handle each response without attaching other response. – Ahsan aslam Mar 04 '20 at 10:26
*I have no idea where to place callback function* - you could call it at the end of `make_request` or at the end of `generate_requests`, depending on what you want the callback to observe. Or am I somehow misunderstanding what you want to achieve? – user4815162342 Mar 04 '20 at 10:30
The above code is working fine but the problem is that in infinite-loop cycle my code wait for each cycle(for country in countries:) generated requests and when all response for one cycle completed it then generates next cycle requests. I want a mechanism in which code should not wait for each cycle response. A callback function should automatically trigger when a response come from post request. – Ahsan aslam Mar 04 '20 at 10:41
@Ahsanaslam Sorry, I don't understand what you mean by "infinite-loop cycle". Can you update the question with the code that better reflects your actual use case? – user4815162342 Mar 04 '20 at 12:13
I am using while True an infinite-loop. And one cycle means when "for country in countries:" completed. – Ahsan aslam Mar 04 '20 at 12:18
@Ahsanaslam But you can add the same `while True` in `generate_all_countries`. And you can still invoke a function of your choosing (or one provided by the caller) inside `make_request` or `generate_requests`. What additional "trigger" do you need? – user4815162342 Mar 04 '20 at 12:24

Get aiohttp request response in callback with a infinite loop of asyncio task

1 Answers1