1

I'm attempting to use aws s3 sync to push a bunch of backup files from my Windows server to an s3 bucket. The problem that I'm running into is that it appears to be picking up the Recycle Bin and System Volume Information. Unfortunately, it's unable to traverse into those directories and so ultimately the command exits with a non-zero status.

Here's what I'm trying as my command line:

aws s3 sync X:\Backup\ s3://«my bucket name» --exclude '*\$RECYCLE.BIN\*' --exclude "System Volume Information\*"  --exclude '*.trn' --include '*.bak'

I've tried several variations of the --exclude parameter but to no avail.

Ben Thul
  • 3,024
  • 17
  • 24
  • The easiest option here is not to back up the whole disk from the root, but to put your data into a single folder with as many subfolders as are required, and syncing only that. – Tim Apr 17 '19 at 01:29
  • Sure. The entire purpose of the drive though is hold backup files. So while I could artificially put another directory in the path so I'm not syncing off of the root, I'd rather not if it's avoidable. – Ben Thul Apr 17 '19 at 19:59
  • I have used exclude in the past, but I'm not near those resources so I can't tell you how. You could look up the docs for file paths and try absolute paths. I know for CloudFormation each command varies for the file format required but often uses file:// type local paths. – Tim Apr 17 '19 at 20:04

2 Answers2

2

Assuming you have dropped to a DOS shell to run that command, single quotes are not recognized as parameter wrappers. Using double quotes worked for me.

aws --profile MyProfile s3 sync X:\ s3://some-bucket/data/ --exclude "*\$RECYCLE.BIN\*" --exclude "*\System Volume Information\*"
Rex Henderson
  • 121
  • 2
0

So for those still struggling with this, here is a partial solution/explanation that may help you get your scripts working. AWS Sync doesn't seem to exclude folders, if it has no access to read that folder, as in the case of the System Volume Information folder. The simplest way to get this working is to grant your script runner access to the folder in question, then the --exclude command should work properly. If that wasn't enough of a kick in the pants, one failing --exclude command will make all subsequent (and maybe preceding) exclude commands fail. So for instance if you tried this:

aws sync $path $bucket --exclude "$path*System Volume System Volume Information*" --exclude '*.trn'

But your user has no access to the sys vol path, it won't even bother to evaluate the *.trn. So I see two possible solutions to this, one is to grant your user account access to the problem folder. Second is to scan your root directory into an array and eliminate those undesirable folders at the array variable level, then for each loop your way through syncing the rest. Messy, but it would get the job done without manually stating each folder in your code.

Hope that helps.

Jon
  • 1