Stack Overflow
Stack Overflow

Questions tagged [s3cmd]

S3CMD QUESTIONS MUST BE PROGRAMMING RELATED. s3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

S3CMD QUESTIONS MUST BE PROGRAMMING RELATED. s3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

See the s3cmd homepage for more information.

282 questions
1
vote
2 answers

sync between 2 s3 buckets using s3cmd not working

I am trying to sync 2 s3 buckets. Here is the command I am using to sync between 2 s3 buckets. s3cmd sync s3://source-bucket s3://destination-bucket I am setting this in the crontab. I have specified the absolute path of the s3cmd. I am logging the…
Jeevan Dongre
  • 4,627
  • 13
  • 67
  • 129
1
vote
1 answer

Deploy gzip content on Amazon S3

I use gzip compression for Amazon S3. I gzip HTML, JS, CSS files and keep images unchanged. I sync everything using s3cmd: s3cmd sync --cf-invalidate ./deploy s3://n12v.com Unfortunately, this doesn’t set Content-Encoding: gzip to all necessary…
NVI
  • 14,907
  • 16
  • 65
  • 104
1
vote
0 answers

How to Solve MagicException: "could not find any magic files" in s3cmd?

I am using s3cmd to upload files. and it always upload .png file as "image/x-png" for MIME type. So I decide to install "python-magic" What I did here: Installed Python 2.7 x86 on Windows 7 64bit, (since the manual of "python-magic" said only x86…
Eric Yin
  • 8,737
  • 19
  • 77
  • 118
1
vote
1 answer

Copying files in the same Amazon S3 bucket

I am trying to copy the entire /contentstore/ folder on a bucket to a timestamped version. Basically /contenstore/ would be copied to /contentstore/20130729/. My entire script uses s3s3mirror first to clone my production S3 bucket to a backup. I…
akshtray
  • 161
  • 1
  • 1
  • 4
1
vote
2 answers

Calling s3cmd with django-celery

I have these lines within a django application. They run fine under normal conditions, but when I call them as a celery task nothing seems to happen. I can't even find an error reported. Could anyone suggest where I could be going wrong? cmd =…
Chris Wheadon
  • 840
  • 6
  • 17
1
vote
2 answers

Running S3cmd from PHP Script not working

I want to use s3cmd from my PHP script. Everything is working from shell but same not working from my PHP script. shell_exec('s3cmd --config=/root/s3cmd.conf ls'); this is not working, and then i gave full path to my s3cmd…
Asghar
  • 2,336
  • 8
  • 46
  • 79
1
vote
1 answer

S3 errors for files greater than 5 GB, uploaded via s3cmd 1.1.0 beta

I'm using s3cmd 1.1.0beta to upload files that are larger than 5 GB to Amazon S3. This is because s3cmd older than 1.1.0 is not able to upload files larger than 5 GB (Amazon single-part upload limit), and the latest beta version is able to upload…
Suman
  • 9,221
  • 5
  • 49
  • 62
1
vote
1 answer

S3Cmd fails as www-data and works as root

I am running s3cmd on Ubuntu as root and it works fine, no issues. But as www-data which is the user my cron uses it fails and I get this. /var/backups/archives/files/file_name_27-01-2013_01:00:27.tar.gz ->…
Jake N
  • 10,535
  • 11
  • 66
  • 112
1
vote
1 answer

s3cmd not "Getting" the distcp jar file

Hi guys : I'm trying to get the s3 distcp jar file via s3, in an EMR cluster : s3cmd get s3://eu-west-1.elasticmapreduce/libs/s3distcp/1.0.1/s3distcp.jar However, the "get" command is not working: ERROR: Skipping libs/s3distcp/: No such file or…
jayunit100
  • 17,388
  • 22
  • 92
  • 167
0
votes
1 answer

S3 Bash Script looping

I've written a script to get the size of my S3 storage, the script works fine barring the fact that it keeps looping and never ends. I get all the outputs as I need but it keeps going through them. Any ideas on why its looping? #!/bin/sh DAY=$(date…
Grimlockz
  • 2,541
  • 7
  • 31
  • 38
0
votes
2 answers

s3sync on ruby 1.9.2 not working

This is a long running issue which I have faced, not being able to use s3sync and s3cmd from ruby ver 1.9.2, although it works fine under ruby 1.8.7. Tried making the changes as give here lib/HTTPStreaming.rb is missing the line: require…
Sam Wilder
  • 869
  • 1
  • 9
  • 21
0
votes
0 answers

Is there an S4CMD version of --no-mime-magic

I am using S3CMD successfully and the --no-mime-magic flag to have it use the same content-type as original source. However S4CMD runs faster and I'd prefer to use it. I don't see any equivilant flag of --no-mime-magic though. Is there one maybe…
wharfdale
  • 1,148
  • 6
  • 23
  • 53
0
votes
0 answers

Broken pipe error while uploading large multipart file via s3cmd: "[Errno 32]" Broken pipe

I'm uploading a large (~200GB) compressed file with s3cmd and keep getting a periodic error. First we try to upload a part, it fails at 0%. Then I get the error: "Cannot retrieve any response status before encountering an EPIPE or ECONNRESET…
myfatson
  • 354
  • 2
  • 14
0
votes
0 answers

Issue with s3cmd accessing DigitalOcean Spaces storage service due to configuration file issue

TL;DR If your use of s3cmd on Spaces from DigitalOcean says: Could not refresh role Then try removing the cfg for it and starting again. I was using the DigitalOcean Spaces and I somehow got my configuration of the API messed up when using the…
mlhamatms
  • 1
0
votes
0 answers

how to set content-disposition header on linode bucket or file?

I am trying to to use s3cmd to set a content-disposition header for images in a Linode storage bucket(Similar to s3 on AWS). These images are used on a website however, are delivered with the following header: Content-Type:…
noob
  • 1
1 2 3
…
18 19