0

I am relatively new to AWS s3 I am calling an API to load the JSON data directly to s3 bucket. From s3 bucket data will be read by Snowflake. After researching I found that using Boto3 we can load data into s3 directly. Code will look something like below, however one thing I am not sure about is What should I put for the key as there is no object created in my S3 bucket. Also, what is the good practice to load the JSON data to s3 ? Do I need to encode JSON data to 'UTF-8' as done here by SO user Uwe Bretschneider.

Thanks in advance!

Python code:

import json,urllib.request
import boto3
data = urllib.request.urlopen("https://api.github.com/users?since=100").read()
output = json.loads(data) 
print (output) #Checking the data


s3 = boto3.client('s3')
s3.put_object(
     Body=str(json.dumps(data))
     Bucket='I_HAVE_BUCKET_NAME'
     Key='your_key_here'
)
biggboss2019
  • 220
  • 3
  • 8
  • 30
  • 1
    Key is analogous to path. It's just a name you use to reference the data in the bucket, it can be whatever you like and you don't need to create a parent path in S3. So you could specify 'a/b/c/d.json' without 'a/b/c' previously existing. – NullPointerException May 21 '22 at 15:52

1 Answers1

2

By using put_object, which means you are creating a new object in the bucket, so there is no existing key. This key is just like a file name in the file system. You can specify whatever names you like, such as my-data.json or some-dir/my-data.json. You can find out more in https://docs.aws.amazon.com/AmazonS3/latest/userguide/object-keys.html.

As for encoder, it's always good to specify the encoding IMO, just to make sure your source file has properly encoded too.

Greg
  • 1,363
  • 13
  • 15
  • Thank you! Where should I put encoder in my above code ? I am receiving "TypeError: Object of type 'bytes' is not JSON serializable" – biggboss2019 May 21 '22 at 16:03
  • 1
    Played with the code for a while, think you can use `output` directly: `output = json.loads(data.decode('utf-8'))` ... `Body=str(json.dumps(output ))` see if that helps, could ignore my previous comment. – Greg May 21 '22 at 16:52