39

I have 4 lambda functions which will be invoked at same time (by SNS), the frequence of SNS's event is 5 minutes. Each function process the large mount of data and images(~300MB) so I store them on /tmp folder (500MB limit).

At the beginning of function, I wrote some code to the clean up /tmp folder, to make sure it's not out of memory (Because I've known that AWS lambda sometimes reuses previous container to improve performance).

I check it manually (create message and publish by SNS to 4 lambda functions), it worked fine.

But when it runs automatically (invoked each 5 minutes) the result is not as my expectation. The first execution is fine, but the next times after, 1 of 4 or even 4 lambda functions throw out the error related to "out of memory": "No space left on device", cannot load lib, ...

Previous, I use nodejs(4.3) it worked fine both case.

But I have to change to python for some reason, the main flow and the mount of created data is the same. But it's failed when run automatically.

I think that the issue came from the cache of previous container (reused container), I checked the /tmp after clean (ls -alh /tmp) there's no files but when check the storage (df /tmp) it show that used is 77%.

Any suggestion to make clean /tmp folder or work around solution is very appreciate. Thank!

Edited: Code I use to clean /tmp folder:

from subprocess import call
...
call('rm -rf /tmp/*', shell=True)
Ngoan Tran
  • 1,507
  • 1
  • 13
  • 17

4 Answers4

25

Yes, lambda being a managed service; they do reuse the same underlying resource if the lambda is getting invoked repeatedly. This was a production problem we faced and fixed by deleting the /tmp. On a separate note AWS should mention this in their FAQs. 

if os.path.exists(tmp_file_path):
        os.remove(tmp_file_path)
        print("Removed the file %s" % tmp_file_path)     
else:
    print("Sorry, file %s does not exist." % tmp_file_path)
Sid
  • 261
  • 3
  • 5
  • 4
    They do mention it in the FAQ: "To improve performance, AWS Lambda may choose to retain an instance of your function and reuse it to serve a subsequent request, rather than creating a new copy. To learn more about how Lambda reuses function instances, visit our [documentation](http://docs.aws.amazon.com/lambda/latest/dg/lambda-introduction.html). Your code should not assume that this will always happen." – bgdnlp Jun 02 '19 at 16:06
3

I tried to replicate this issue using lambdash, a great function to test out commands in your "dev" account. It allows you to run arbitrary UNIX commands in the Lambda environment.

I ran this command repeatedly, and did not see the issue appear. Note: The commands are not actually in the deployed code so this test does not fully replicate the potential issue.

lambdash "echo Checking:;file /tmp/nullfile;rm -f /tmp/nullfile;df -h /tmp;dd if=/dev/zero bs=1024 count=88888 >> /tmp/nullfile; echo ==========;df -h /tmp"
Mike Behr
  • 43
  • 4
2

Containers are often reused, but not concurrently. Clean up your temp directory when the function finishes and see if issue resolves.

omuthu
  • 5,948
  • 1
  • 27
  • 37
  • I've tried it, I cleaned up /tmp at the beginning and the end in code. – Ngoan Tran May 22 '17 at 10:52
  • 1
    I confirm this bug on node 10 containers today. I do remove all /tmp files (and dirs) but FREE disk space is shrinking after each invoke – kjetildm Sep 18 '18 at 12:06
1

Warning rm -rf /tmp/* don't remove all files (hidden files or dir will not be deleted).

For a real clean of /tmp dir you should use : rm -rf /tmp/..?* /tmp/.[!.]* /tmp/*

For a complete explanation read: https://unix.stackexchange.com/a/77313/66488

PiR
  • 175
  • 9