Stack Overflow
Stack Overflow

Questions tagged [s3cmd]

S3CMD QUESTIONS MUST BE PROGRAMMING RELATED. s3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

S3CMD QUESTIONS MUST BE PROGRAMMING RELATED. s3cmd is a command line tool for uploading, retrieving and managing data in Amazon S3. It is best suited for power users who don't fear command line. It is also ideal for scripts, automated backups triggered from cron, etc.

See the s3cmd homepage for more information.

282 questions
2
votes
1 answer

Change permissions

I use s3cmd to upload files to Amazon s3. It is a super application. The only problem is I still need to log into the web interface and change the permissions of the file to "Authorized Users" and grant Read on it. Is there a way I can change the…
user983223
  • 1,146
  • 2
  • 16
  • 32
2
votes
1 answer

How do you pipe the result of s3cmd get to a var?

I need to get an image from S3, process it on an ec2, and save it to a different folder on S3. I would like to not save the file to ec2 during the process. Is it possible to pipe the result of "s3cmd get s3://bucket/image" to a var? It does not…
waigani
  • 3,570
  • 5
  • 46
  • 71
2
votes
1 answer

What is the most efficient S3 GET request method?

I can download a file from S3 using either of the following methods. s3cmd get s3://bucket_name/DB/company_data/abc.txt wget http://bucket_name.s3.amazonaws.com/DB/company_data/abc.txt My question is : 1) Which one is faster? 2) Which one is…
shantanuo
  • 31,689
  • 78
  • 245
  • 403
1
vote
1 answer

Chaining terminal script on mac os x

I am trying to chain some terminal commands together so that i can wget a file unzip it and then directly sync to amazon s3. Here is what i have so far i have s3cmd tool installed properly and working. This works for me. mkdir extract; wget…
DCHP
  • 1,111
  • 6
  • 29
  • 53
1
vote
0 answers

How do you configure s3cmd for Windows 10/Python 3.10?

I'm using Windows 10 and trying to find a way to access my private S3 repository. I have GitBash and this version of Python $ python --version Python 3.10.5 I downloaded s3cmd v2.2 but when after running the set up process successfully, I get this…
Dave
  • 15,639
  • 133
  • 442
  • 830
1
vote
1 answer

s3cmd using incorrect MIME type for .js files

I recently upgraded my laptop and my s3cmd script to version 2.2.0 but my *.js files that used to upload with Content-Type "application/javascript" now use "text/plain". Here's the command I've been using: python s3cmd --access_key=…
Ben
  • 917
  • 9
  • 14
1
vote
1 answer

How can I list all empty folders with s3cmd

I have uploaded 5k+ folders, each of which should have one file and one subfolder (this subfolder then holds various files) to DigitalOcean Spaces (S3 storage). It looks like some of the uploads failed (long story). Is there a way to list all empty…
eskimo
  • 2,421
  • 5
  • 45
  • 81
1
vote
0 answers

s3cmd put files from tar stdin: [Errno 32] Broken pipe

I was trying to upload files from stdin by s3cmd, by this command below tar cfz - folder | s3cmd put - s3://backups/abcd2.tar but sometimes I get ERROR ERROR: Cannot retrieve any response status before encountering an EPIPE or ECONNRESET…
user2971916
  • 61
  • 8
1
vote
1 answer

"s3cmd get" for only changed files

I need to run the s3cmd get command every day and only download the changed files (if any). There's this --skip-existing option which skips the already existing files but it doesn't check whether the file on S3 has changed in contents or not. Is…
Marry Jane
  • 305
  • 2
  • 15
1
vote
2 answers

Why is Module python-magic not available?

I am using s3cmd to sync to an S3 bucket from a Mac and I am getting the following warning: WARNING: Module python-magic is not available. Guessing MIME types based on file extensions. But pip list shows that python-magic is in fact…
Mark Fisher
  • 965
  • 1
  • 11
  • 30
1
vote
1 answer

How to upload multiple files from directory to S3?

I have a directory on an Ubuntu, with 340K images, 45GB of total size! Is there an efficient way to transfer them all to an S3 of DigitalOcean? I thought of using s3cmd put or s3cmd sync but I'm guessing that would perform the put operation on every…
Sotiris Kaniras
  • 520
  • 1
  • 12
  • 30
1
vote
0 answers

Upload compressed files with s3cmd guessing mime-types correctly

I've a large number of compressed small files (~10k, that have {txt,html,log,xml,binary}.gz extension) to be sync-ed/put into S3 buckets regularly, created with public https read access with the s3cmd. These are mostly files with different…
askb
  • 6,501
  • 30
  • 43
1
vote
0 answers

Invalidating index.html with s3cmd

I'm using s3cmd in my CI/CD deploy pipeline. My files are uploaded to a bucket in S3, which is then internet-accesible from CloudFront. After reading AWS's docs carefully, I understand that CloudFront will cache, by default, all files from S3,…
alexandernst
  • 14,352
  • 22
  • 97
  • 197
1
vote
0 answers

Download files from Amazon S3 windows command line utility

I am tasked to automate the process of downloading files from Amazon S3. I am quite new this, after searching on the web I found this command line utility for windows s3.exe I run the get command with the below syntax s3 get xxx-xxx/yyyy_yyy/ D:\…
sab
  • 338
  • 1
  • 3
  • 21
1
vote
2 answers

Which is more secure S3cmd or AWS s3 (AWS CLI)?

My team currently use s3cmd command to transfer files from Unix to S3 bucket. However, our IT has raised security concerns for using the cmd. I know AWS s3 is another alternative but wanted to know if it is secure? Thanks in advance
Anurag Chand
  • 19
  • 6
1 2 3
…
18 19