Stack Overflow
Stack Overflow

Questions tagged [s3cmd]

S3CMD QUESTIONS MUST BE PROGRAMMING RELATED. s3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

S3CMD QUESTIONS MUST BE PROGRAMMING RELATED. s3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

See the s3cmd homepage for more information.

282 questions
0
votes
1 answer

Access to S3 object but not bucket

I have been given full access to an S3 folder by means of an ACL. However, when I try to list the objects in that folder I get an ERROR: Access to bucket '[bucket name]' was denied I do not have access to the bucket. I was only given access to the…
Bayan Bruss
  • 1
0
votes
1 answer

Search & count total instances of a specific pattern in an s3 bucket without downloading the file

I need to count the number of lines with a matching patterns across s3 buckets. The command I am using is -: s3cmd ls --recursive s3://mys3.com/bucket1/ | awk '{print $4}' | grep '.lzo' | xargs -I@ s3cmd get @ - | zgrep 'my-pattern-of-interest-1' |…
ekta
  • 1,560
  • 3
  • 28
  • 57
0
votes
2 answers

How to set up scheduled backups in amazon s3?

I have S3 bucket "foo.backups", s3cmd is installed on my DigitalOcean droplet. I need to backup mydb.sqlite3 database and "myfolder". How to make scheduled daily backups of these databse and folder with such…
NonGrate
  • 862
  • 9
  • 22
0
votes
2 answers

Is there any way to get list of folder names in amazon s3 bucket?

I am writing PHP script for uploading folders if not found in amazon S3 bucket using s3cmd . For that first i need to get all folder names and store it in array and then check array using array_search if folder name is there or not and proceed to…
maheshiv
  • 1,778
  • 2
  • 17
  • 21
0
votes
1 answer

Call S3Express in batch, pass command, loop

I've got a .bat file that will apply a few quick ImageMagick conversions across a large collection of subdirectories. I need to get this script to also push the output files to AWS S3 using the S3Express utility (Windows variant of S3cmd). echo…
muad-dweeb
  • 995
  • 1
  • 9
  • 24
0
votes
1 answer

add expires header with bash scripts returns incorrect entry on AWS with s3cmd

i am using this bash script to push files to S3: #!/bin/bash for f in $(find nas/cdn/catalog/drawings \( ! -regex '.*/\..*' \) -type f) do extension=`file $f | cut -d ' ' -f2 | awk '{print tolower($0)}'` mimetype=`file --mime-type $f |…
khinester
  • 3,398
  • 9
  • 45
  • 88
0
votes
2 answers

bash script to ittarate trough files and push to s3

i have the following bash script: s3upload() { n=$1 extension=`file $n | cut -d ' ' -f2 | awk '{print tolower($0)}'` mimetype=`file --mime-type $n | cut -d ' ' -f2` fullpath=`readlink -f $n` expires="Expires:`date -u +"%a, %d %b…
khinester
  • 3,398
  • 9
  • 45
  • 88
0
votes
1 answer

Download NCDC weather dataset from s3 to local machine using S3cmd

I would like to download the publicly available NCDC weather dataset on amazon public datasets. But am unable to find it. Could anyone tell me the exact bucket it is located in? Also could you please tell me how to download it onto my local machine…
Nikhil
  • 545
  • 1
  • 7
  • 18
0
votes
1 answer

Delete incomplete downloads with s3cmd

I'm using s3cmd on a network with bad Internet connection. Sometimes, it stops in the middle of a sync resulting in half finished files/corrupted files. Normally, I'd like it to simply continue where it paused, but in this case due to various…
Paolo
  • 2,161
  • 5
  • 25
  • 32
0
votes
1 answer

Sync with S3 with s3cmd, but not re-download files that only changed name

I'm syncing a bunch of files between my computer and Amazon S3. Say a couple of the files change name, but their content is still the same. Do I have to have the local file removed by s3cmd and then the "new" file re-downloaded, just because it has…
Paolo
  • 2,161
  • 5
  • 25
  • 32
0
votes
1 answer

S3 Bucket Permissions

I have an s3 bucket with the policy { "Version": "2008-10-17", "Id": "Policy1412967758094", "Statement": [ { "Sid": "Stmt1412967757168", "Effect": "Allow", "Principal": { "AWS":…
dacox
  • 502
  • 1
  • 6
  • 14
0
votes
2 answers

In Hive, how do I load only part of the raw data to a table?

I've got a typical CREATE TABLE statement as follows: CREATE EXTERNAL TABLE temp_url ( MSISDN STRING, TIMESTAMP STRING, URL STRING, TIER1 STRING ) row format delimited fields terminated by '\t' lines terminated by '\n' LOCATION…
zack_falcon
  • 4,186
  • 20
  • 62
  • 108
0
votes
2 answers

Commandline tools to restore a versioned deleted file in S3?

I have versioning enabled in my S3 bucket, how do I restore a deleted file with a command line S3 client, such as s3cmd? How do I browse the different versions of the files? So far, I have regressed to Freeware Cloudberry Windows Client to achieve…
John Mayor
  • 577
  • 6
  • 13
0
votes
1 answer

What is Chef doing when I use the `s3cmd` recipe?

I am using Chef and this s3cmd cookbook. As this tutorial says I use knife to download and tar it. I actually made s3cmd work following the tutorial instructions, but I have problems understanding where exactly the installation of s3cmd is…
user2266685
  • 191
  • 1
  • 2
  • 8
0
votes
1 answer

Moving several files from one bucket to another in AWS returns error: Timed Out

I'm using s3cmd to copy over 5000 files from one bucket to a folder within another bucket. Like this: s3cmd mv --recursive -v s3://test.bucket/1111_stuff/ s3://actual.bucket/input/dataloader_input/ However, this keeps getting me this: INFO:…
zack_falcon
  • 4,186
  • 20
  • 62
  • 108
1 2 3
…
18 19