20

I want to copy a set of files over from S3, and put them in the /tmp directory while my lambda function is running, to use and manipulate the contents. The following code excerpt works fine on my PC (which is running windows)

s3 = boto3.resource('s3')
BUCKET_NAME = 'car_sentiment'
keys = ['automated.csv', 'connected_automated.csv', 'connected.csv', 
        'summary.csv']
for KEY in keys: 
    try:
        local_file_name = 'tmp/'+KEY
        s3.Bucket(BUCKET_NAME).download_file(KEY, local_file_name)
    except botocore.exceptions.ClientError as e:
        if e.response['Error']['Code'] == "404":
            continue
        else:
            raise

However, when I try to run on AWS lambda, I get: 

{
  "errorMessage": "[Errno 2] No such file or directory: 'tmp/automated.csv.4Bcd0bB9'",
  "errorType": "FileNotFoundError",
  "stackTrace": [
    [
      "/var/task/SentimentForAWS.py",
      28,
      "my_handler",
      "s3.Bucket(BUCKET_NAME).download_file(KEY, local_file_name)"
    ],
    [
      "/var/runtime/boto3/s3/inject.py",
      246,
      "bucket_download_file",
      "ExtraArgs=ExtraArgs, Callback=Callback, Config=Config)"
    ],
    [
      "/var/runtime/boto3/s3/inject.py",
      172,
      "download_file",
      "extra_args=ExtraArgs, callback=Callback)"
    ],
    [
      "/var/runtime/boto3/s3/transfer.py",
      307,
      "download_file",
      "future.result()"
    ],
    [
      "/var/runtime/s3transfer/futures.py",
      73,
      "result",
      "return self._coordinator.result()"
    ],
    [
      "/var/runtime/s3transfer/futures.py",
      233,
      "result",
      "raise self._exception"
    ],
    [
      "/var/runtime/s3transfer/tasks.py",
      126,
      "__call__",
      "return self._execute_main(kwargs)"
    ],
    [
      "/var/runtime/s3transfer/tasks.py",
      150,
      "_execute_main",
      "return_value = self._main(**kwargs)"
    ],
    [
      "/var/runtime/s3transfer/download.py",
      582,
      "_main",
      "fileobj.seek(offset)"
    ],
    [
      "/var/runtime/s3transfer/utils.py",
      335,
      "seek",
      "self._open_if_needed()"
    ],
    [
      "/var/runtime/s3transfer/utils.py",
      318,
      "_open_if_needed",
      "self._fileobj = self._open_function(self._filename, self._mode)"
    ],
    [
      "/var/runtime/s3transfer/utils.py",
      244,
      "open",
      "return open(filename, mode)"
    ]
  ]
}

Why does it think the file name is tmp/automated.csv.4Bcd0bB9 rather than just tmp/automated.csv and how do I fix it? Been pulling my hair out on this one, trying multiple approaches, some of which generate a similar error when running locally on my PC. Thanks!

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
ViennaMike
  • 2,207
  • 1
  • 24
  • 38

2 Answers2

32

You should save in /tmp, rather than tmp/.

eg:

local_file_name = '/tmp/' + KEY
John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
  • 1
    I would have sweared, and bet money, that I had tried that, but obviously not, since it works now. Maybe it's because adding that extra slash breaks it on my windows machine, But it works now. Will accept answer shortly. And thanks for reformatting the mess. I had just jumped back on the PC to fix it when I saw you already had. Out of curiosity and to learn, why does the garbage characters get added to the end of the file name when I do it incorrectly? – ViennaMike Jul 07 '18 at 00:42
  • I know this is a pretty old answer, but.. why? Is /tmp/ really different to tmp/ don't they both start at the root of my lambda function files? – lorless Apr 04 '19 at 15:05
  • 2
    @lorless The leading `/` indicates an absolute path. Using only `tmp/` will be a relative offset from the "current" directory, which is not `/`. – John Rotenstein Apr 04 '19 at 20:28
  • 3
    Just as a note, you can't do directories in /tmp/. Files have to go directly in /tmp/ – Michael Brant Sep 22 '19 at 01:19
  • 4
    @MichaelBrant I did a test and successfully created a directory in `/tmp/` by using `os.mkdir('/tmp/foo')` within a Lambda function. – John Rotenstein Sep 22 '19 at 05:57
  • @JohnRotenstein You are absolutely right. Thank you for letting me know.. this saved me time! – Michael Brant Sep 23 '19 at 01:31
  • Resolved the error. Thanks a lot. – Geshan Ravindu Sep 26 '21 at 04:31
1

Well the reason why lambda gives the above error is because it doesn't allow to write in the heirarchial structure with in the /tmp/ directory. You can write the files directly to /tmp/example.txt dir but not to say /tmp/dir1/example.txt

kumar gautam
  • 11
  • 3