Using boto3 to await a synchronous lambda invocation

Question

I want to invoke a lambda function synchronously (request - response) but want to use python async-await to await the response.

    response = await client.invoke('my-func', InvocationType='RequestResponse', Payload='...'))

I found a kind of solution here but it is cumbersome and from 2016.

Is there a better approach today?

boto3 does not support asyncio. So unless you put it in a separate thread, it will block. — Marcin, Jul 28 '21 at 09:54
Can you describe what you would expect this to do? If you want the call to be synchronous (which it is), you don’t need `await`. — dirn, Jul 28 '21 at 10:57
@dirn I think the line of code pretty much explains it. Maybe you are confusing python asyncio `await` with lambda synchronous invocation? They use the same terms but they are completely different concepts implemented in completely different levels. — daramasala, Jul 28 '21 at 13:08
But you haven’t explained what that line of code should do. What does it mean to await a synchronous operation? And if it isn’t the `await` that’s associated with asyncio, what `await` is it? The asyncio tag should probably be removed from the question since it’s unrelated. — dirn, Jul 28 '21 at 13:20
@dirn this question is specifically about using asyncio with boto3 and lambda service. The `await` is asyncio await. Behind the scenes `client.invoke` makes an HTTP request. The "synchronous invocation" thing is related to the payload of the request and how lambda works. The question is tagged asyncio AND boto3 because you need to know both in order to answer. — daramasala, Jul 28 '21 at 13:28
boto3 doesn’t work with asyncio. You’ll need to use aioboto3 for that. Otherwise you’ll want to use a `ThreadPoolExecutor` from `concurrent.futures`. — dirn, Jul 28 '21 at 13:33

score 1 · Accepted Answer · answered Jan 31 '22 at 08:17

I found a way of doing it by manually running the invoke function on the asyncio event loop:

import asyncio
import concurrent
import boto3
import json

import botocore


class LambdaClient():
    def __init__(self, concurrency: int = 20):
        self.executor = concurrent.futures.ThreadPoolExecutor(
            max_workers=concurrency,
        )
        client_config = botocore.config.Config(
           max_pool_connections=concurrency
        )
        self.client = boto3.client('lambda', config=client_config)
    
    async def invoke_async(self, snapshot):
        loop = asyncio.get_running_loop()
        result = await loop.run_in_executor(self.executor, lambda: self.invoke(snapshot))
        return result

    def invoke(self, snapshot):
        payload = {
        'path': '/calculate/value',
        'body': json.dumps(snapshot)
        }
        b = bytes(json.dumps(payload), encoding='utf8')
        response = self.client.invoke(
            FunctionName='function-name',
            InvocationType='RequestResponse',
            LogType='None',
            Payload=b)
        if 'StatusCode' not in response or response['StatusCode'] != 200:
            raise ValueError(f'Lambda invocation failed with response {response}')
        output = response["Payload"].read()
        return output

Using boto3 to await a synchronous lambda invocation

1 Answers1