0

I have a script to download files using asyncio. The problem I am facing is when an exception is rised, it is catched within the task but the task is resumed and start the loop again

In downloader a loop for pagination has been implmented, the inner loop attemps stop after 3 attemps and to stop the outer loop pagging an exception is rised and catched in except ... At that point the coroutine should return but instead of that the loop attemps start again but this time with a value = 2

This behaivor only happen when several tasks (the coroutne downloader) run at the same time (when several tasks are submitted to the loop) If the coroutine run one by one the the coroutine has the spected behaivour 

async def downloader(session, url, spe_params, path, name, writer,
                     max_features=5000, sleeping=5):
    try:
        pagging = True
        while pagging:
            attemps = 3
            while attemps:
                # attemps reduced for looping
                try:
                    async with session.get(url, params=p) as res:
                        if res.status != 200:
                            raise aiohttp.ClientError
                        res_json = await res.json()

                except (aiohttp.ClientError, asyncio.TimeoutError,
                        json.JSONDecodeError) as e:
                    message = 'Error fetching {0}'.format(str(e))
                    attemps -= 1
                    await asyncio.sleep(sleeping)
                else:
                    if res_json['totalFeatures'] == number_returned:
                        pagging = False
                        break
                    # Here the logic for pagination

            if not attemps:
                # attemps have been consumed
                # Pagination stops and raise exception
                raise DownloadError(message)

    except DownloadError as e:
        # Exception is catched. The coroutine should end but instead of that
        # the loop "attemps" start again with the original value - 1

    else:
        # Json written to file

async def main():
    loop_tasks = []
    for id in field_id:
        task = loop.create_task(downloader(session, ... )

    done, pending = await asyncio.wait(loop_tasks,
                                               return_when='ALL_COMPLETED')
  • *coroutine should return* - I don't see any `return` in your code – RomanPerekhrest Jul 19 '19 at 15:10
  • @RomanPerekhrest return is not needed since there is not more code after the except catching –  Jul 19 '19 at 15:40
  • 1
    this code is so difficult to read! why are you raising exceptions only to immediately catch them? why have those nested while loops? at least you're now using `asyncio.wait`, but why specify the documented default value? also `raise` interrupts execution and hence having anything after it, i.e. the `break`, is pointless – Sam Mason Jul 19 '19 at 15:57

0 Answers0