1

I have a very simple function (py3.6) that I'm running on sam-local via start-api that I'm testing with a shell script full of curl posts:

lambda_handler(event, context):
  try:
    json_body = event["body"]
    if not validate_json(json_body):
      raise Exception("Bad JSON")
    post_to_dynamo_db_local(json_body)
  except Exception as err:
    return {
        'statusCode': 999,
        'headers': {"x-custom-header": "my custom header value"},
        'body': "\nException! " + str(err)
    }
  return {
    'statusCode': 200,
    'headers': {"x-custom-header": "my custom header value"},
    'body': "hello world"
  }

The test works fine in the correct use-case about 95% of the time, but fails (inconsistently) when validate_json raises an Exception despite being caught, or when post_to_dynamo_db_local shorts and returns early (object already in DB, return).
＊I say correct use-case works about 95% of the time, because sometimes I touch stuff and it fails, but me touching stuff is likely the cause. Unconfirmed though!

The function appears to return properly, according to the sam-local console:

START RequestId: 8c7cf7ce-2926-4ce4-ba3c-0fac95a810b0 Version: $LATEST
END RequestId: 8c7cf7ce-2926-4ce4-ba3c-0fac95a810b0
REPORT RequestId: 8c7cf7ce-2926-4ce4-ba3c-0fac95a810b0 Duration: 162 ms 
Billed Duration: 200 ms Memory Size: 128 MB Max Memory Used: 24 MB

But the curl doesn't receive a response until the request times out:

  { "message": "Internal server error" }

and around the same time the sam-local console responds:

  Error invoking python 3.6 runtime: io: read/write on closed pipe

Any ideas? The function run-time seems abnormally high for such a simple task, and I don't think memory should be an issue here...

Failing case flow:
Start -> check input(string) -> raise exception -> catch exception -> return response(string)

ADDITION: A couple things that help:

  • Sleeping between curls (tried up to 10 seconds, but still errors occasionally
  • Not caching references to tables (so they should be collected fairly quickly now
Mars
  • 2,505
  • 17
  • 26
  • lambda is limited to 1024 file descriptors: https://docs.aws.amazon.com/lambda/latest/dg/limits.html. You could be running into this limit if things aren't being properly closed. – avigil Feb 20 '18 at 02:39
  • Seems that if that were true I'd get an `exceeds limit` exception rather than the closed pipe one. My function really is as simple as described above, so it's not likely that I'm hitting the any limits... but is there any manual closing that I should be doing in my lambda functions? – Mars Feb 20 '18 at 02:44
  • https://stackoverflow.com/questions/47089665/aws-lambda-nodejs-runtime-io-read-write-on-closed-pipe suggests timeout as the culprit. Try increasing timeout or memory/cpu allocation – avigil Feb 21 '18 at 04:40
  • @avigil Thanks, but the timeout is already very high. The function returns, hence printing out the lambda function usage REPORT, but something hangs in the backend and the http response is never sent – Mars Feb 21 '18 at 04:49
  • I think `Error invoking python 3.6 runtime` means something is not cleaning up after itself and the container running your lambda is eventually exhausting its resources after repeated executions. See if there is anything you might want to put into a `finally` block of your exception handler or use a context manager for anything that might be opening a file descriptor – avigil Feb 21 '18 at 05:02
  • @avigil Nothing to put in a finally block and zero file descriptors being used in the code. It's really odd that two successful cases run back to back, but a successful case followed by a bad-json/quick return raises the error. – Mars Feb 21 '18 at 05:25
  • It doesn't seem to be a timing issue either though--adding a wait between curls doesn't solve the issue – Mars Feb 21 '18 at 05:34

0 Answers0