0

Im facing difficulties while trying to execute around 10 functions simultaneously using asyncio in Django. I could not find nay clear documentation on how to use asyncio with django.

I made around 10 http requests to different using TOR which is by default slow. Instead of making these requests one by one, which usually takes around 2 minutes, I wanted to perform all the 10 requests simultaneously. There are 10 distinct functions each making a http request a different URL and scrape data and return JSON.

URLscrape.py :

async def scrape1(username):
    response = request.get('http://example.com'+username)
    return response.json()

async def scrape2(username):
    response = request.get('http://something.com'+username)
    return response.json()

I have 10 individual functions like the above with different URLs and perfrom scraping and return a json data. In Django views, I did like this:

Views.py

from URLscrape import scrape1, scrape2........scrape10

def scrapper():
    loop = asyncio.get_event_loop()
    feature1 = loop.run_in_executor(None, scrape1, username)
    feature2 = loop.run_in_executor(None, scrape2, username)
    ....
    feature10 = loop.run_in_executor(None, scrape10, username)

    response1 = await future1
    response2 = await future2 
    .....
    response10 = await future10
 
    response1 = response1.text
    ......
    response10 = response2.text
  
    return render(request, 'index.html', {'scrape1':response1,'scrape10':response10})

But I dont know how to use loop.run_until_complete() to complete the script. Im in a restricted situation to use 10 individual functions for scraping. Im want to run the 10 simultaneously but I dont know how. I could not understand the concept and syntax of asyncio. Please help!!!

DisappointedByUnaccountableMod
  • 6,656
  • 4
  • 18
  • 22
Adithyan AK
  • 1
  • 1
  • 1
  • You probably need to drop the awaits and get the responses with `response1, response2, ... = loop.run_until_complete(asyncio.gather(feature1, feature2, ...))` – user4815162342 Jul 11 '20 at 21:27
  • Also, using asyncio just to run `requests` under `run_in_executor` doesn't bring you any of the benefits of asyncio, you're still running threads under the hood. I would recommend switching to an asyncio-aware http library such as `aiohttp`. – user4815162342 Jul 11 '20 at 21:28

0 Answers0