Stack Overflow
Stack Overflow

Questions tagged [boto3]

Boto 3 - The Amazon Web Services (AWS) SDK for Python

About

Boto is the Amazon Web Services (AWS) software development kit (SDK) for Python, which allows Python developers to write software that makes use of services like Amazon S3 and Amazon EC2. Boto provides an easy to use, object-oriented API as well as low-level direct service access.

Links

7627 questions
2
votes
1 answer

How to upload the folder to digital ocean spaces?

From the link, https://www.digitalocean.com/community/questions/how-to-upload-an-object-to-digital-ocean-spaces-using-python-boto3-library. It only states to upload files to the spaces. I want to upload a folder to the spaces. import boto3 session…
Atom Store
  • 961
  • 1
  • 11
  • 35
2
votes
0 answers

Is there an api available to configure/trigger or create new scheduled incremental refreshes for AWS Quicksight datasets?

I have a requirement where I need to refresh my dataset programmatically every x minutes. Now this value of x could change based on some other parameters like how frequently new data is coming or based on user activity on dashboards, decided by…
Akhil Raj
  • 21
  • 2
2
votes
0 answers

Aws boto3 redshift client wait until sql executes

client = boto3.client('redshift-data') # this seems asynchronous response = client.execute_statement( ClusterIdentifier='xyz', Database='dev', DbUser='user1', Sql='select * from table1;', StatementName='test1', …
Aseem
  • 5,848
  • 7
  • 45
  • 69
2
votes
1 answer

find last modified date of a particular file in S3

According to this answer: https://stackoverflow.com/a/9688496/13067694 >>> import boto >>> s3 = boto.connect_s3() >>> bucket = s3.lookup('mybucket') >>> for key in bucket: print key.name, key.size, key.last_modified index.html 13738…
x89
  • 2,798
  • 5
  • 46
  • 110
2
votes
1 answer

how to specify ContentType for S3 files?

I am trying to convert files to gzip files and then upload them to S3. When I check in S3, the files are there but they don't have a type specified. How can I specify the content type? for i in testList: with contextlib.ExitStack() as…
user13067694
2
votes
1 answer

Error when retrieving credentials from Environment: No credentials found in credential_source referenced in profile

Env file: AWS_ACCESS_KEY_ID=AAAAAAAAAAAAAAAAAA AWS_SECRET_ACCESS_KEY=xxxxxxxxxxxxxxxxxxxxxxxxxxxx AWS_DEFAULT_REGION=eu-west-1 AWS_DEFAULT_OUTPUT=json Config file: [profile a] credential_source = Environment region=eu-west-1 role_arn=arn [profile…
mrc
  • 2,845
  • 8
  • 39
  • 73
2
votes
0 answers

AWS CLI Download Performance Far Faster than Boto3

I'm attempting to download a significant number of small files from AWS S3 (50,000+) and I'm consistently noting that the AWS CLI sync command is dominating my solution written in boto3. I've started looking through the AWS CLI source for ways my…
FlamboyantBacon
  • 41
  • 5
2
votes
2 answers

How are credentials refreshed with cached boto3 client and Lambda Provisioned Concurrency?

Following the best practice Take advantage of execution environment reuse to improve the performance of your function, I am investigating if caching the boto3 client has any negative effect while using Lambda Provisioned Concurrency. The boto3…
Giulio Micheloni
  • 1,290
  • 11
  • 25
2
votes
2 answers

How to concurrently list_objects in S3 bucket with aioboto3

I want to find all the unique paths in an S3 bucket, I want all the paths just before the file level. The depth of the directories can vary so not all files are found in the same depth for example I might have these files…
KZiovas
  • 3,491
  • 3
  • 26
  • 47
2
votes
0 answers

Python: boto3 decrypt

I'm trying to decrypt via the boto3 a ecrypted value with kms. The value is encrypted via the aws cli and then stored: aws kms encrypt --key-id xxx --plaintext "test" The code that I'm trying to use to decrypt then is following: kms_response =…
Yannick Tresch
  • 21
  • 1
2
votes
2 answers

list_schemas() method missing on Boto3 Glue client object

So, I think I'm running up against an issue with out of date documentation. According to the documentation here I should be able to use list_schemas() to get a list of schemas defined in the Hive Data Catalog:…
Adam Luchjenbroers
  • 4,917
  • 2
  • 30
  • 35
2
votes
1 answer

Iterate through S3 objects, rather than just all of the key /bucket in an object

Below is a commonly shared function to iterate over all objects in a bucket, but what if I just want to iterate over a specific key ie lets say the S3 URI was: s3://test-data-lake/test1/test2/ And there was five json files after test two ie…
0004
  • 1,156
  • 1
  • 14
  • 49
2
votes
1 answer

listing s3 buckets using boto3 and python

I am using the below code and referred to many SO answers for listing files under a folder using boto3 and python but was unable to do so. Below is my code: s3 = boto3.client('s3') object_listing = s3.list_objects_v2(Bucket='maxValue', …
whatsinthename
  • 1,828
  • 20
  • 59
2
votes
1 answer

Put (Lambda) subscription on CW Logs group

I am trying to put a subscription on a CW log group from a Lambda Function that is scanning for lambdas with the right tag. When calling the put_subscription_filter an Error is thrown: "An error occurred (InvalidParameterException) when calling the…
Erik Asplund
  • 673
  • 4
  • 14
2
votes
0 answers

Test S3 operations with simulated credentials

I'm writing unit tests for a function that should authenticate to AWS S3 and then perform some operations on S3. I have a bunch of functions that do various things (like downloading/uploading files, checking for existence, etc). Since in production…
cyau
  • 449
  • 4
  • 14
1 2 3
…
99
100