Writing json to file in s3 bucket

Question

This code writes json to a file in s3, what i wanted to achieve is instead of opening data.json file and writing to s3 (sample.json) file,

how do i pass the json directly and write to a file in s3 ?

import boto3

s3 = boto3.resource('s3', aws_access_key_id='aws_key', aws_secret_access_key='aws_sec_key')
s3.Object('mybucket', 'sample.json').put(Body=open('data.json', 'rb'))

Are you saying that you want to pass JSON data directly to a file that sits on S3 without having to upload a new file to s3? — Usman Mutawakil, Oct 20 '17 at 07:46
yes... in simple.. i get some json data after some operations, i want to create a filename.json file in S3 and write that json to this file. — Learning, Oct 20 '17 at 07:53

score 97 · Answer 1 · answered Oct 26 '18 at 07:58

97

I'm not sure, if I get the question right. You just want to write JSON data to a file using Boto3? The following code writes a python dictionary to a JSON file.

import json
import boto3    
s3 = boto3.resource('s3')
s3object = s3.Object('your-bucket-name', 'your_file.json')

s3object.put(
    Body=(bytes(json.dumps(json_data).encode('UTF-8')))
)

answered Oct 26 '18 at 07:58

Uwe Bretschneider

1,211
8
12

1

does the 'your-bucket-name' include the path through any subfolders in the bucket or does that go into 'your-file.json' name? – Sledge Dec 16 '19 at 14:56
5

@Sledge - Not in the bucket, the filename contains that information: `s3.Object('your-bucket-name', 'folder/subfolder/your_file.json')` – robertlayton Dec 16 '19 at 22:59
4

You should also specify a content type. If you're using this method, you should add the ContentType param such as: s3object.put( Body=(bytes(json.dumps(json_data).encode('UTF-8'))), ContentType='application/json' ) – Alex Young Jun 11 '20 at 18:51

score 73 · Answer 2 · edited Jul 07 '21 at 23:25

73

I don't know if anyone is still attempting to use this thread, but I was attempting to upload a JSON to s3 trying to use the method above, which didnt quite work for me. Boto and s3 might have changed since 2018, but this achieved the results for me:

import json
import boto3

s3 = boto3.client('s3')
json_object = 'your_json_object here'
s3.put_object(
     Body=json.dumps(json_object),
     Bucket='your_bucket_name',
     Key='your_key_here'
)

edited Jul 07 '21 at 23:25

Alexander Santos

1,458
11
22

answered Jul 10 '20 at 17:27

Joseph McCombs

731
5
3

6

why use `str()`? `json.dumps()` (plural dumps) creates a JSON formatted string. – pjw Dec 22 '20 at 00:44
`Body (bytes or seekable file-like object) -- Object data.`. `json.dumps` will return `str`, not bytes. You'll need to `.encode("utf-8")`. From https://boto3.amazonaws.com/v1/documentation/api/latest/reference/services/s3.html#S3.Client.put_object – Alberto Megía May 13 '22 at 12:38

score 7 · Accepted Answer · answered Oct 20 '17 at 07:56

7

Amazon S3 is an object store (File store in reality). The primary operations are PUT and GET. You can not add data into an existing object in S3. You can only replace the entire object itself.

For a list of available operations you can perform on s3 see this link http://docs.aws.amazon.com/AmazonS3/latest/API/RESTObjectOps.html

answered Oct 20 '17 at 07:56

Usman Mutawakil

4,993
9
43
80

i get this error - botocore.exceptions.ClientError: An error occurred (PermanentRedirect) when calling the ListObjects operation: The bucket you are attempting to access must be addressed using the specified endpoint. Please send all future requests to this endpoint. – Learning Oct 20 '17 at 08:00
See if this helps https://stackoverflow.com/questions/32246149/permanentredirect-when-calling-the-putobject-operation – Usman Mutawakil Oct 20 '17 at 08:03
So you have to replace the whole file each time you want to change the data in it? – lolelo Jul 18 '21 at 13:58
1

@lolelo Yep. If you use the REST API or the SDK thats using the rest api then the whole file is replaced. I'm not aware of an alternate solution. – Usman Mutawakil Jul 20 '21 at 14:22

score 3 · Answer 4 · answered Jul 19 '22 at 22:04

An alternative to Joseph McCombs answer can be achieved using s3fs.

from s3fs import S3FileSystem

json_object = {'test': 3.14}
path_to_s3_object = 's3://your-s3-bucket/your_json_filename.json'

s3 = S3FileSystem()
with s3.open(path_to_s3_object, 'w') as file:
    json.dump(json_object, file)

Writing json to file in s3 bucket

4 Answers4

Linked