a bucket is one of several alternative data sinks when data needs to be distributed. The term is used for algorithms, data-structures. Commercial usages include databases and cloud providers.
Questions tagged [bucket]
993 questions
11
votes
1 answer
What is the difference between partitioning and bucketing in Spark?
I try to optimize a join query between two spark dataframes, let's call them df1, df2 (join on common column "SaleId").
df1 is very small (5M) so I broadcast it among the nodes of the spark cluster.
df2 is very large (200M rows) so I tried to…
nofar mishraki
- 526
- 1
- 4
- 15
11
votes
9 answers
How to fix AccessDenied calling CopyObject
I'm trying to copy files from a bucket in A account to another bucket but in B account. When I try to sync the files with the command
aws s3 sync s3://BUCKET_A s3://BUCKET_B
It returns the following output:
copy failed: s3://BUCKET_A to…
Carlos Andres
- 12,740
- 7
- 18
- 34
11
votes
1 answer
S3: An error occurred (AccessDenied) when calling the GetObject operation: Access Denied
After configure AWSCLI using command aws configure . I'm trying to download files from S3 bucket to local folder test, using following command
aws s3 sync s3://sourceBucket ./test --delete
but it's throwing following error message
download failed:…
Lakshmi
- 125
- 1
- 1
- 6
11
votes
1 answer
Why Doesn't My AWS S3 Bucket Policy Override My IAM Policy?
I have a user in my IAM account called "testuser" who has administrator privileges, like so:
{
"Statement": [
{
"Effect": "Allow",
"Action": "*",
"Resource": "*"
}
]
}
And then I have a policy on my S3 bucket that…
Dasmowenator
- 5,505
- 5
- 36
- 50
10
votes
2 answers
Find S3 Bucket Owner
If there is a bucket that I have read/write access on from my AWS account, is there any way to get the CanonicalUser id of the owner?
Reading the ACL Policy seems to be impossible unless you are the owner : /.
Felix Geisendörfer
- 2,902
- 5
- 27
- 36
10
votes
2 answers
I need to configure CORS in s3 bucket the code I have isn't a valid JSON
I am trying to make a photo gallery from my s3 photo bucket. the next step in my process is to configure CORS but when I use the code provided I get an error. Can someone please explain what I am doing wrong or how to change the code to valid…
Skyler James Fagan
- 101
- 1
- 1
- 4
10
votes
1 answer
How do I upload a CSV file in myBucket and Read File in S3 AWS using Python
How do I upload a CSV file from my local machine to my AWS S3 bucket and read that CSV file?
bucket = aws_connection.get_bucket('mybucket')
#with this i am able to create bucket
folders = bucket.list("","/")
for folder in folders:
print…
TB.M
- 363
- 3
- 8
- 26
10
votes
4 answers
AWS S3 Bucket with Multiple Regions
I have an AWS S3 bucket which is set for the US Standard region but I want it to work in Singapore too. I have researched but could not find a way for the same bucket to work i multiple regions.
It would be great if someone could find a…
Maanit
- 316
- 2
- 4
- 14
9
votes
7 answers
How do I delete/count objects in a s3 bucket?
So I know this is a common question but there just doesn't seem to be any good answers for it.
I have a bucket with gobs (I have no clue how many) number of files in them. They are all within 2k a piece.
1) How do I figure out how many of these…
eyberg
- 3,160
- 5
- 27
- 43
9
votes
2 answers
How to show the bucket name when using RANGE_BUCKET in bigquery
Here is my query with public dataset in BigQuery:
SELECT RANGE_BUCKET(reputation, [400000, 500000, 600000, 700000, 800000, 900000, 1000000, 1100000, 1200000]) AS reputation_group, COUNT(*) AS count
FROM…
mdivk
- 3,545
- 8
- 53
- 91
9
votes
4 answers
How can I convince spark not to make an exchange when the join key is a super-set of the bucketBy key?
While testing for a production use-case I have created and saved (using Hive Metastore) such tables:
table1:
fields: key1, key2, value1
sortedBy key1,key2
bucketBy: key1, 100 buckets
table2:
fields: key1, key2, value2
sortedBy: key1,key2
bucketBy:…
zetaprime
- 278
- 2
- 14
9
votes
2 answers
AWS Bucket Policy Error: Policy has invalid action
I have a very basic goal: to share all content of my bucket to a list of specific users, read only. This used to work with a tool called s3cmd. All I need to do was to add a user (identified by email) to the Access Control List with Read Permission,…
fanchyna
- 2,623
- 7
- 36
- 38
9
votes
1 answer
What is a bucket or double-bucket data structure?
I'm doing some reading about shortest path algorithm implementations and have been running into over and over that implementing Dijkstra's Algorithm with a Double-Bucket data structure is a good implementation.
However I cannot seem to find what a…
Carson
- 1,147
- 5
- 19
- 41
9
votes
4 answers
I am studing couchbase, can anyone exlain what exactly is bucket and vbucket?
I am studing couchbase now, I am really confused by the official description of the term 'bucket' and 'vbucket', can anybody explain what exactely a bucket or vbucket is ? what's the difference?
Better to make some analogies and give some examples.
user3012468
- 93
- 1
- 3
8
votes
3 answers
Change user ownership of s3fs mounted buckets
how can I modify the user:group ownership of a s3fs mounted bucket?
I have a git installation that I would essentially like to store on my Amazon S3 account in a bucket, and then using Sparkleshare, via my web host, sync this data accross multiple…
itsricky
- 387
- 1
- 3
- 9