To push the file from lambda to s3

Question

I have triggered to lambda with the kinesis stream and looking for a record where action is blocked and have appeneded the data to output file.

how can i push that file to s3 ?I have written below but not sure.

New code import json

import urllib.parse
import boto3

print('Loading function')


s3 = boto3.client('s3')

def lambda_handler(event, context):

    #1 - Get the bucket name
    bucket = event['Records'][0]['s3']['bucket']['name']

    #2 - Get the file/key name
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')


    #3 - Fetch the file from S3
    response = s3.get_object(Bucket=bucket, Key=key)

    #4 - Deserialize the file's content
    text = response["Body"].read().decode()
    e  = text.split("\n")

    Output=[]
    #5 - Print the content
    print(text)

    #6 - Parse and print the Action

    for each in e:
        loaded_data = json.loads(e)
        if loaded_data["action"] == "ALLOW":
            print("dropped")
        else :    
            Output.append(loaded_data)
    s3.put_object(Body='json.dumps(output)',Bucket='blocketreques',Key='Filtered.txt')
    print('Put Complete')

import json
import urllib.parse
import boto3

print('Loading function')


s3 = boto3.client('s3')

def lambda_handler(event, context):

    #1 - Get the bucket name
    bucket = event['Records'][0]['s3']['bucket']['name']

    #2 - Get the file/key name
    key = urllib.parse.unquote_plus(event['Records'][0]['s3']['object']['key'], encoding='utf-8')


    #3 - Fetch the file from S3
    response = s3.get_object(Bucket=bucket, Key=key)

    #4 - Deserialize the file's content
    text = response["Body"].read().decode()
    e  = text.split("\n")

    Output=[]
    #5 - Print the content
    print(text)

    #6 - Parse and print the Action

    for each in e:
        loaded_data = json.loads(e)
        if loaded_data["action"] == "ALLOW":
            print("dropped")
        else :    
            Output.append(loaded_data)
    s3.put_object(Body='json.dumps(output)',Bucket='blocketreques',Key='Filtered.txt')
    print('Put Complete')

`if message["action']` watch out for typos; `if message["action"]` — andreis11, Apr 07 '20 at 13:21
Also note wrong indentation near `if message["action']=="BLOCKED":` — Vsevolod Skripnik, Apr 08 '20 at 06:43

score 0 · Accepted Answer · answered Apr 08 '20 at 06:31

0

The code is using s3.upload_file(), which uploads a file from disk.

If you want to upload the contents from memory (eg the output array), you can use:

s3.put_object(Body=json.dumps(output), Bucket=..., Key=...)

answered Apr 08 '20 at 06:31

John Rotenstein

241,921
22
380
470

Hi Thanks for point out , i have made the changes but it gives me error inconsistent use of tabs and spaces in indentation (lambda_function.py, line 36).i have update the code in question – chandni mirchandani Apr 09 '20 at 07:29
You need to indent the contents of `else:`. Also, you didn't close the quotes in the `print()` statement. (That's why the colours look wrong.) Oh, and `Output[]` should be `Output = []`. The error messages are normally good indicators that something is wrong. – John Rotenstein Apr 09 '20 at 09:38
now i get below error :Response: { "errorMessage": "An error occurred (NoSuchKey) when calling the GetObject operation: The specified key does not exist.", "errorType": "NoSuchKey", "stackTrace": [ [ "/var/task/lambda_function.py", 20, "lambda_handler", "response = s3.get_object(Bucket=bucket, Key=key)" ], [ "/var/runtime/botocore/client.py", 316, "_api_call", "return self._make_api_call(operation_name, kwargs)" ], [ – chandni mirchandani Apr 09 '20 at 10:24
`The specified key does not exist` means that there is no object with the given Key (filename) in the bucket you specified. You will need to debug your code to figure out why (eg print the Bucket and Key to have them appear in CloudWatch Logs, then check whether that file actually exists). – John Rotenstein Apr 09 '20 at 11:15

score 0 · Answer 2 · answered Jul 17 '20 at 13:04

I recommend to just use smart-open

It behaves just like any other opened file but if you give it s3 path, it will save it to s3. Also it will handle other scenarios like multi part uploads if you are trying to save bigger files.

PS: When testing locally you can just change output path to point to local machine and speed up your testing

To push the file from lambda to s3

2 Answers2