3

how can I write my results from a file processing step with AWS lambda and python back to a file? I'm reading a file from S3 and looking for a special expressing in each line. If this expression is included, I manipulate the line. As lambda is not able to write to a file (or S3 does not allow this), how can a collect the result of the line transformation and write all transformed line into a file? the code looks like this:

import boto3
import botocore

s3 = boto3.resource('s3')
s3 = boto3.client('s3')

def lambda_handler(event, context):

bucket = event['Records'][0]['s3']['bucket']['name']
key = event['Records'][0]['s3']['object']['key']

obj = s3.get_object(Bucket=bucket, Key=key)

for line in obj['Body'].read().decode('utf-8').splitlines():
    if 'PCSI' in line:
        newLine = line \
        .replace('E','') \
        .replace('--','') \
        .replace('<',';') \
        .replace('>','') \
        .replace('9_PCSI','') \
        .replace('[','') \
        .replace('|',';') \
        .replace(']',';') \
        .replace(' ','')

when I print the results it works fine and gives me the format of each line I want.

One idea I had was to write into a file in the /tmp folder of lambda:

newFile = open('/tmp/pcsi.txt','a')

and modify the code like

...
if 'PCSI' in line:
    newFile.write(line \
    .replace(.....

but I do not know if this works as I can not "see" the file in the /tmp. I also struggled with downloading it again to S3. Is there a way to write each line into a file and store it to S3?

John Rotenstein
  • 241,921
  • 22
  • 380
  • 470
Kebre
  • 33
  • 1
  • 4
  • 1
    Lambdas are able to write to S3 buckets, you just neet to use `boto3` and have the proper IAM settings. – yorodm Dec 14 '18 at 16:53
  • An alternative method would be to simply **download** the file to `/tmp`. Then you can read it like a normal file and do whatever you wish. If you wish to save a modified file back to S3, simply write to a local file, then **upload** it to S3. – John Rotenstein Dec 15 '18 at 04:01
  • download will not work, everything should run in AWS without local PCs – Kebre Dec 19 '18 at 08:35

1 Answers1

2

You are halfway there. You have read the object from s3 and manipulated it to your requirement. Now, you have to write it to s3 so that the modifications that you made in the lambda function are reflected in s3.

This can be accomplished by

Here, using put_object would look like

client.put_object(Body=manipulated_object, Bucket=bucket, Key=key)
Jojin
  • 124
  • 1
  • 8
  • put is clear - thanks. I struggle to write all manipulated lines into the file as there are several in the original file with the "PCSI" expression. Lambda and S§ do not allow an append case as I understood it. With the put option I can write the last manipulated line to S3 but not all?! Therefore I thought to write in an append case the data to /tmp of lambda and than this new file to S3 but no success so far (or would there be another way?) – Kebre Dec 19 '18 at 09:57