Stack Overflow
Stack Overflow

Questions tagged [boto3]

Boto 3 - The Amazon Web Services (AWS) SDK for Python

About

Boto is the Amazon Web Services (AWS) software development kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2. Boto provides an easy to use, object-oriented API as well as low-level direct service access.

Links

7627 questions
128
votes
8 answers

Mocking boto3 S3 client method Python

I'm trying to mock a singluar method from the boto3 s3 client object to throw an exception. But I need all other methods for this class to work as normal. This is so I can test a singular Exception test when and error occurs performing a…
ptimson
  • 5,533
  • 8
  • 35
  • 53
127
votes
4 answers

Error "Read-only file system" in AWS Lambda when downloading a file from S3

I'm seeing the below error from my lambda function when I drop a file.csv into an S3 bucket. The file is not large and I even added a 60 second sleep prior to opening the file for reading, but for some reason the file has the extra ".6CEdFe7C"…
user1530318
  • 25,507
  • 15
  • 37
  • 48
127
votes
18 answers

Boto3 to download all files from a S3 Bucket

I'm using boto3 to get files from s3 bucket. I need a similar functionality like aws s3 sync My current code is #!/usr/bin/python import boto3 s3=boto3.client('s3') list=s3.list_objects(Bucket='my_bucket_name')['Contents'] for key in list: …
Shan
  • 2,141
  • 2
  • 17
  • 32
125
votes
3 answers

How to get more than 1000 objects from S3 by using list_objects_v2?

I have more than 500,000 objects on s3. I am trying to get the size of each object. I am using the following python code for that import boto3 bucket = 'bucket' prefix = 'prefix' contents = boto3.client('s3').list_objects_v2(Bucket=bucket, …
tahir siddiqui
  • 1,371
  • 2
  • 8
  • 8
119
votes
10 answers

Read file content from S3 bucket with boto3

I read the filenames in my S3 bucket by doing objs = boto3.client.list_objects(Bucket='my_bucket') while 'Contents' in objs.keys(): objs_contents = objs['Contents'] for i in range(len(objs_contents)): filename =…
mar tin
  • 9,266
  • 23
  • 72
  • 97
104
votes
2 answers

When to use a boto3 client and when to use a boto3 resource?

I am trying to understand when I should use a Resource and when I should use a Client. The definitions provided in boto3 docs don't really make it clear when it is preferable to use one or the other.
aquil.abdullah
  • 3,059
  • 3
  • 21
  • 40
103
votes
10 answers

Complete scan of dynamoDb with boto3

My table is around 220mb with 250k records within it. I'm trying to pull all of this data into python. I realize this needs to be a chunked batch process and looped through, but I'm not sure how I can set the batches to start where the previous…
CJ_Spaz
  • 1,134
  • 2
  • 7
  • 10
101
votes
3 answers

Boto3/S3: Renaming an object using copy_object

I'm trying to rename a file in my s3 bucket using python boto3, I couldn't clearly understand the arguments. can someone help me here? What I'm planing is to copy object to a new object, and then delete the actual object. I found similar questions…
MikA
  • 5,184
  • 5
  • 33
  • 42
99
votes
7 answers

Reading an JSON file from S3 using Python boto3

I kept following JSON in S3 bucket 'test' { 'Details' : "Something" } I am using following code to read this JSON and printing the key 'Details' s3 = boto3.resource('s3', aws_access_key_id=, …
Nanju
  • 1,011
  • 1
  • 7
  • 9
93
votes
11 answers

s3 urls - get bucket name and path

I have a variable which has the aws s3 url s3://bucket_name/folder1/folder2/file1.json I want to get the bucket_name in a variables and rest i.e /folder1/folder2/file1.json in another variable. I tried the regular expressions and could get the…
Lijju Mathew
  • 1,911
  • 6
  • 20
  • 26
89
votes
5 answers

How to upload a file to S3 and make it public using boto3?

I am able to upload an image file using: s3 = session.resource('s3') bucket = s3.Bucket(S3_BUCKET) bucket.upload_file(file, key) However, I want to make the file public too. I tried looking up for some functions to set ACL for the file but seems…
Adi
  • 4,149
  • 4
  • 25
  • 41
88
votes
1 answer

How to assume an AWS role from another AWS role?

I have two AWS account - lets say A and B. In account B, I have a role defined that allow access to another role from account A. Lets call it Role-B { "Version": "2012-10-17", "Statement": [ { "Effect": "Allow", "Principal": { …
Prashant
  • 3,823
  • 3
  • 25
  • 40
84
votes
8 answers

How to check if DynamoDB table exists?

I'm a new user in boto3 and i'm using DynamoDB. I went through over the DynamoDB api and I couldn't find any method which tell me if a table is already exists. What is the best approach dealing this issue? Should I try to create a new table and…
roeygol
  • 4,908
  • 9
  • 51
  • 88
84
votes
2 answers

Is it possible to combine if_not_exists and list_append in update_item

I'm trying to use the update_item functionality for DynamoDB in boto3. I'm struggling right now to update lists for items. I would like to create a new list if the list does not exist yet and otherwise append to the existing list. Using an…
fabian
  • 883
  • 1
  • 6
  • 6
81
votes
14 answers

Unable to install boto3

I have trouble installing boto3 inside a virtual environment. I have done what the document says. First I activated virtual environment. then I did a: Sudo pip install boto3 Now I enter python >> import boto3 ImportError: No module named…
Prabhakar Shanmugam
  • 5,634
  • 6
  • 25
  • 36
1
2
3
…
99 100