0

My azure function reads a url from a queue and writes it to blob store. Though it is burning through the queue very quickly, I am not seeing the blobs in blob store. My guess is that I am doing something wrong with async and the function is finishing too early.

Before I added the async calls this was working correctly.

Note* I am using azure.storage.blob.aio so the blob storage call is async.

I'm surprised because I'm following the documentation pretty closely:

https://learn.microsoft.com/en-us/azure/azure-functions/python-scale-performance-reference#async

The main difference between my use case and the documentation is that the docs have one async operation (https request) and my usecase has two asyncs (https request and blob storage save).


import logging

import azure.functions as func
from azure.identity import DefaultAzureCredential
from azure.storage.blob.aio import BlobServiceClient
import aiohttp

async def main(msg: func.QueueMessage) :

    url:str = msg.get_body().decode('utf-8')
    logging.info(url)

    account_url = "https://SOMETHING.blob.core.windows.net"
    default_credential = DefaultAzureCredential()
    blob_service_client = BlobServiceClient(account_url, credential=default_credential)

    file_name = url.replace('https://', '')
    blob_client = blob_service_client.get_blob_client(container='chords', blob=f'{file_name}.txt')

    async with aiohttp.ClientSession() as client:
        async with client.get(url) as response:
            await blob_client.upload_blob(await response.text())

Maybe the issue is that I have two awaits on the last line?.

await blob_client.upload_blob(await response.text())

Thanks all.

Initially I tried without the async blob update, the code worked but was very slow. I tried without the 'await' before the blob client, but that also did not work. I also tried not having async at all but the code executed far too slowly as it is IO bound.

jrudd
  • 113
  • 2
  • I can't see anything obviously wrong in your code. Two awaits on the last line is fine. I would debug if I were you. Preferably locally, with breakpoints, or with the logging module on Azure (unless VSCode supports remote debugging, I'm not sure). – FiddleStix Feb 08 '23 at 17:38

1 Answers1

0

I found two things wrong with my code.

Adding the 'await' to the next line and replacing 'azure.storage.blob' with 'azure.storage.blob.aio' seemed to do the trick!

jrudd
  • 113
  • 2