3

Synchronous invocation: If the function is invoked synchronously and is throttled, Lambda returns a 429 error and the invoking service is responsible for retries. The ThrottledReason error code explains whether you ran into a function level throttle (if specified) or an account level throttle (see note below). Each service may have its own retry policy. For example, CloudWatch Logs retries the failed batch up to five times with delays between retries. For a list of event sources and their invocation type, see Supported Event Sources.

Reference

I not sure my understanding about the above sentence is right, If I am wrong please fix me.

  1. When a lambda is throttled, it returns 429 Error to API-gateway.
  2. The invoking service, in here API-gateway, retries the request.

However, It doesn't work as expected. The below is the API-gateway log from cloudWatch when a lambda is throttled.

API-Gateway-Execution-Logs_3f1frvtwe4/sam-sm-test 2a38a4a9316c49e5a833517c45d31070 (bededbf0-73ae-11e8-87a2-f51933ef104f) Endpoint response body before transformations: {"Reason":"ReservedFunctionConcurrentInvocationLimitExceeded","Type":"User","message":"Rate Exceeded."}
API-Gateway-Execution-Logs_3f1frvtwe4/sam-sm-test 2a38a4a9316c49e5a833517c45d31070 (bededbf0-73ae-11e8-87a2-f51933ef104f) Endpoint response headers: {Connection=keep-alive, x-amzn-RequestId=bedfc624-73ae-11e8-bd28-6345cb3606c4, x-amzn-ErrorType=TooManyRequestsException, Content-Length=104, Date=Tue, 19 Jun 2018 10:51:39 GMT, Content-Type=application/json}
API-Gateway-Execution-Logs_3f1frvtwe4/sam-sm-test 2a38a4a9316c49e5a833517c45d31070 (bededbf0-73ae-11e8-87a2-f51933ef104f) Execution failed due to configuration error: Malformed Lambda proxy response

In practical, Lambda returns {"Reason":"ReservedFunctionConcurrentInvocationLimitExceeded","Type":"User","message":"Rate Exceeded."} which is wrong format for API-gateway(proxy integration) then, as the result, API-gateway returns 502 Error to client calling the API.

I want the failed request to be retried. how could I handle it?

Community
  • 1
  • 1
SangminKim
  • 8,358
  • 14
  • 69
  • 125
  • The service may retry the request, depending on the retry policy of that specific service. You are making an assumption about the API Gateway retry policy which I believe is incorrect. – Mark B Jun 19 '18 at 13:43

1 Answers1

9

Each service may have its own retry policy.

API Gateway will not retry a failed invocation of a Lambda. If you want to handle a retry, this will have to be done in the client calling the API Gateway.

A 502 error is, as you suggested, returned by the API Gateway when it receives a malformed Lambda proxy response (see https://aws.amazon.com/premiumsupport/knowledge-center/malformed-502-api-gateway/).

K Mo
  • 2,125
  • 8
  • 16
  • Thank you for your answering!!. One more question, even if API-gateway doesn't handle retry, Why lambda doesn't return 429Error? Seeing API-gateway log, the lambda seems just to return JSON string not containing statusCode. – SangminKim Jun 19 '18 at 14:26
  • 2
    @SangminKim That is just one of the pains of a lambda proxy integration. API Gateway expects a response from lambda in a very specific format (see https://docs.aws.amazon.com/apigateway/latest/developerguide/set-up-lambda-proxy-integrations.html#api-gateway-simple-proxy-for-lambda-output-format). The actual http status is not captured (or at least visible) to API Gateway unless it is stated in the lambda response. The 429 error does not return a response in this format. – K Mo Jun 19 '18 at 15:04
  • @KMo Lambda DLQ is not applicable to synchronous Lambda invocations. The caller is always responsible for retrying 429 errors. Only repeated async failures after the invocation request is initially accepted by Lambda will result in an event going to the DLQ. – Michael - sqlbot Jun 20 '18 at 11:52
  • @Michael - sqlbot Reference to DLQ removed – K Mo Jun 20 '18 at 12:09
  • @KMo do you have a link to documentation for "API Gateway will not retry a failed invocation of Lambda"? I'd like to read more on this. – totsubo Feb 19 '21 at 01:06