Server Fault Stack Exchange
Server Fault Stack Exchange

Questions tagged [s3cmd]

s3cmd is a simple command-line interface to storing and retrieving data to/from Amazon's S3 cloud storage service.

43 questions
2
votes
1 answer

How to copy all s3 buckets to another AWS account?

I'm copying all EC2 instances to another AWS account (creating the AMIs, sharing them with that account and running the instances from AMIs there). I also need to move my S3 buckets since the applications at EC2 instances are using that buckets to…
Eugene Naydenov
  • 121
  • 1
  • 5
2
votes
1 answer

How backup files with logrotate and Amazon S3?

I would like to create a new standard of two things: how long would be generated and be rotated the logs generated by my applications ? how to transfer the logs to Amazon S3, as a backup server ?. I was thinking in use logrotate, to rotate and…
Valter Silva
  • 155
  • 1
  • 4
  • 14
2
votes
0 answers

Swift Stack Proxy Server Reboot

I'm fairly new to Object Storage and have been thrown into support for an old product we no longer have a support contract for and therefore can't seek help from the vendor. I have a 13-node cluster, 10 object nodes, and 3 proxy nodes. I'm wondering…
user16411104
  • 31
  • 1
1
vote
1 answer

Gitlab not backing up to s3. Where can I find logs?

I am currently trying to migrate our old sameersbn/gitlab:12.1.6 installation to a new Gitlab instance set up with the helm chart. The easiest way seems to enable the s3 backup capability on the old installation, connect the new instance to the same…
Moritz Schmitz v. Hülst
  • 181
  • 1
  • 6
1
vote
0 answers

Spectra S3 Java CLI (DS3) difficulty verifying checksum

We have a SpectraLogic BlackPearl+T950 which uses S3 style commands to put/get/sync etc. To access the Spectra flavour of S3 commands we use a Java CLI client to perform these operations. Supposedly there's an option to validate…
Smock
  • 141
  • 7
1
vote
1 answer

How to copy more than 300GB files one AWS EC2 server to another EC2

I need to copy more than 300 GB files (most them are images) one EC2 instance to another EC2 instance, what will be the best method for doing this? EBS Snapshot will be not work in my situation because I will be need only one directory copy.
Mi2
  • 13
  • 4
1
vote
1 answer

Ways to extract files information for large data

I am trying to recursively extract some file information in my file server. I was able to run the commands below at my own laptop. When I run them on my file server, which is going thru 4TB of data, they run or stuck for hours. When I use the…
hello
  • 11
  • 2
1
vote
1 answer

s3cmd error on sync: S3 error: 400 (InvalidRequest)

I'm trying to configure s3cmd to backup some directories on my server to my AWS S3 bucket. My problem is that when I do the sync command: s3cmd sync -r -p --no-delete-removed /path/to/dir s3://bucketname/dir/ I get the following error: ERROR: S3…
Bart Bergmans
  • 633
  • 1
  • 6
  • 9
1
vote
1 answer

Getting error : Malformed URL error in ec2

I am getting the error Malformed URL ec2-describe-instances instance_id xxx Malformed URL: 'ec2.ap-southeast-1.amazonaws.com' However, I am able to get access to the bucket through command line: s3cmd ls s3://edgeproductionlb-logs DIR …
Ashish Karpe
  • 277
  • 2
  • 5
  • 19
1
vote
1 answer

s3 service using s3cmd tool through URL based firewalls

I am behind a URL based application firewall and currently using s3cmd tool to access amazon's s3 service. Doing a bit of searching has asked me to whitelist s3.amazonaws.com but I see that s3cmd uses a custom URL which goes by s3://[bucket-name].…
lordlabakdas
  • 111
  • 3
1
vote
1 answer

Redirect warning when running s3cmd sync

When running s3cmd I get a redirect warning as follows: WARNING: Redirected to: xyz.s3-eu-west-1.amazonaws.com The script that produces the warning is this: s3cmd sync /var/backups/db/ s3://xyz/database/ I've used this script before elsewhere…
Aron
  • 189
  • 7
1
vote
1 answer

Setting cache for already uploaded amazon s3 files

In amazon s3 for our site through s3cmd tool we uploaded around 60GB of files including mp3, jpg to a bucket. But now when we checked site slowness files are not caching and further checking we came to know that we can set…
GEO PC
  • 31
  • 2
  • 5
1
vote
1 answer

Saving httpd logfiles to S3 on shutdown using s3cmd

I have an auto-scaling group on Amazon Web Services (AWS). I want to preserve the logfiles on an instance (Amazon Linux AMI) when the group scales down. My current idea is to use the s3cmd tool in a script that runs when the instance is rebooted or…
unknownrisk
  • 113
  • 3
1
vote
1 answer

One Way Sync of a Bucket With Local Directory

I have a local directory that I would like to synchronize with an S3 bucket. I have two specific requirements: If local file is the same as the remote, do not re-transfer it to the bucket. If some files or directories exist in the bucket but do not…
Artium
  • 131
  • 6
1
vote
3 answers

Backing up data (including mysqldumps) to S3

We have a web app on a number of servers and we want to add an additional layer of redundancy by backing up the key data to S3. The key data is the MySQL database and a folder containing dynamically created site assets - predominantly images. Some…
robjmills
  • 990
  • 9
  • 26
1
2
3