Stack Overflow
Stack Overflow

Questions tagged [s3cmd]

S3CMD QUESTIONS MUST BE PROGRAMMING RELATED. s3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

S3CMD QUESTIONS MUST BE PROGRAMMING RELATED. s3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

See the s3cmd homepage for more information.

282 questions
14
votes
4 answers

Uploading files to s3 using s3cmd in parallel

I've got a whole heap of files on a server, and I want to upload these onto S3. The files are stored with a .data extension, but really they're just a bunch of jpegs,pngs,zips or pdfs. I've already written a short script which finds the mime type…
Alan Hollis
  • 1,292
  • 3
  • 14
  • 29
14
votes
6 answers

How to find zero byte files in Amazon S3

Is there a way to programmatically find zero bytes file in Amazon S3? The total size of the bucket is more than 100G, unlikely for me to sync back to server, then do a find . -size 0 -type f
ajreal
  • 46,720
  • 11
  • 89
  • 119
12
votes
6 answers

Granting read access to the Authenticated Users group for a file

How do I grant read access to the Authenticated Users group for a file? I'm using s3cmd and want to do it while uploading but I'm just focusing directly on changing the acl. What should I put in for…
user983223
  • 1,146
  • 2
  • 16
  • 32
12
votes
1 answer

S3 Sync Not Writable Warning

I am trying to use s3cmd sync (vanilla command remote to local), but for each file I get a not writable: Operation not permitted warning (see below). The files appear to download correctly, however, and if I do an ls on the file it is there. I tried…
ui_90jax
  • 759
  • 2
  • 6
  • 17
12
votes
2 answers

S3cmd sync returns "killed"

I am trying to sync a few large buckets on amazon S3. When I run my S3cmd sync --recursive command I get a response saying "killed". What does this refer to? Is there a limit on the number of files that can be synced in S3?
Spyros Lambrinidis
  • 253
  • 4
  • 8
11
votes
2 answers

Exclude folders for s3cmd sync

I am using s3cmd and i would like to know how to exclude all folders within a bucket and just sync the bucket root. for example bucket folder/two/ folder/two/file.jpg get.jpg with the sync i just want it to sync the get.jpg and ignore the folder…
user1503606
  • 3,872
  • 13
  • 44
  • 78
10
votes
3 answers

Can I move an object into a 'folder' inside an S3 bucket using the s3cmd mv command?

I have the s3cmd command line tool for linux installed. It works fine to put files in a bucket. However, I want to move a file into a 'folder'. I know that folders aren't natively supported by S3, but my Cyberduck GUI tool converts them nicely for…
Nic Cottrell
  • 9,401
  • 7
  • 53
  • 76
9
votes
1 answer

s3cmd sync is remote copying the wrong files to the wrong locations

I've got the following as part of a shell script to copy site files up to a S3 CDN: for i in "${S3_ASSET_FOLDERS[@]}"; do s3cmd sync -c /path/to/.s3cfg --recursive --acl-public --no-check-md5 --guess-mime-type --verbose…
Wintermute
  • 2,973
  • 4
  • 32
  • 52
9
votes
3 answers

Backing up to S3: symlinks

I'm trying to figure out right now how to backup some data to S3. We have a local backup system implemented using rsnapshot and that works perfectly. We're trying to use s3cmd with the --sync option to mimic rsync to transfer the files. The problem…
Iain Mckay
  • 198
  • 1
  • 5
8
votes
4 answers

Automatically sync two Amazon S3 buckets, besides s3cmd?

Is there a another automated way of syncing two Amazon S3 bucket besides using s3cmd? Maybe Amazon has this as an option? The environment is linux, and every day I would like to sync new & deleted files to another bucket. I hate the thought of…
novice18
  • 89
  • 1
  • 1
  • 4
8
votes
2 answers

How to change storage class of object in s3 bucket?

I need to change storage class of a s3 object from STANDARD class into STANDARD_IA using s3cmd. Can anybody help me to do this task. Thanks in advance.
Gugan Abu
  • 546
  • 1
  • 4
  • 17
7
votes
3 answers

S3Cmd doesn't work with S3 Ninja

I'm trying to use my local s3ninja with s3cmd. Every command like: s3cmd ls s3://test throws the same exceptions. The s3cfg seems to be ok and the called endpoints are correct. Was anyone able to use s3ninja with s3cmd? PS: I know S3 isn't costly…
Daniel R.
  • 507
  • 2
  • 7
  • 21
7
votes
2 answers

How to set the profile when running the s3cmd sync command

When I use aws sync, I can set the profile by doimg aws sync ./test s3://test --profile stage The profile matches to a set of credentials. I have different credentials for each profile. How can I use the s3cmd sync command and set a profile as I do…
Chris Hansen
  • 7,813
  • 15
  • 81
  • 165
7
votes
6 answers

Access Denied s3cmd from an EC2 machine

I'm trying to use a log rotation configuration for my nginx server that I'm using as a reverse proxy machine located on an EC2 Ubuntu instance. I want to store those logs on a S3 bucket after a rotation but I'm only getting "access denied, are you…
Sam
  • 418
  • 2
  • 6
  • 18
7
votes
1 answer

S3 redirect 302 object with s3cmd

I wish to create redirect object. It is redirect to JSON file. HTML expires header will not work for me. touch google s3cmd --add-header='Location: http://www.google.com/' -P put google s3://bucket/google It does not work as I do not set HTTP…
Eugene
  • 257
  • 1
  • 6
1
2
3
…
18 19