6

We have an aws s3 sync command that sync's the data from bitbucket to s3 via Jenkins execute shell. Below is the s3 bucket structure

cdn-accountname > qa > sitename > cdn > img
cdn-accountname > qa > sitename > cdn > css
cdn-accountname > qa > sitename > cdn > png

We use below command to sync the cdn content from bitbucket to s3

aws s3 sync cdn/ s3://cdn-accountname/qa/sitename

I would like to exclude "/upload" and "/upload_qa" folders within cdn/ folder and sync it to s3. I tried below commands but none of them worked

aws s3 sync --exclude "cdn/upload" --exclude "cdn/upload_qa" cdn/ s3://cdn-accountname/qa/sitename
aws s3 sync --exclude=cdn/upload --exclude=cdn/upload_qa cdn/ s3://cdn-accountname/qa/sitename
aws s3 sync --exclude "*/upload/*" --exclude "*/upload_qa/*" cdn/ s3://cdn-accountname/qa/sitename
aws s3 sync --exclude 'upload/*' --exclude 'upload_qa/*' cdn/ s3://cdn-accountname/qa/sitename 
aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude 'upload/*' --exclude 'upload_qa/*'
aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude "cdn/upload" --exclude "cdn/upload_qa"
aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude=cdn/upload --exclude=cdn/upload_qa
aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude "*/upload/*" --exclude "*/upload_qa/*"

suggest the workable s3 sync command to sync the folder excluding sub-folder

MichealMills
  • 315
  • 5
  • 15
  • Is there a particular error you're getting? – maafk Feb 12 '18 at 18:51
  • There will be no errors. When I run the sync command then both upload and upload_qa folders within cdn folder are getting synced to s3. But the expected output should be that they should be excluded from the sync and should not be copied to the s3 bucket. – MichealMills Feb 13 '18 at 10:52
  • 1
    The command `aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --exclude "upload/*" --exclude "upload_qa/*"` should work just fine. Are you sure `/upload` and `/upload_qa` are being synced every time? Maybe they got synced initially and remained there after adding the `--exclude` switch. Did you try to delete them from S3 before syncing? – Khalid T. Feb 21 '18 at 08:37
  • Yes, Every time I running the sync command those files and synced to s3 and then I go back there and delete those files, and try again. But none of the above sync commands worked. – MichealMills Feb 21 '18 at 13:12
  • Try running: `aws s3 sync cdn/ s3://cdn-accountname/qa/sitename --include "*" --exclude "upload/*" --exclude "upload_qa/*"` – DonJuma Feb 26 '18 at 21:41
  • still not working. Let me check directly with AWS support team. – MichealMills Mar 01 '18 at 10:59
  • For now, I am using aws rm command to remove both uploads and uploads_qa folders after the aws sync is done. That is: `aws s3 sync cdn/ s3://cdn-accountname/qa/sitename aws s3 rm --recursive s3://cdn-accountname/qa/sitename/uploads aws s3 rm --recursive s3://cdn-accountname/qa/sitename/uploads_qa` Let me know if someone could help on this – MichealMills Mar 01 '18 at 12:31

1 Answers1

3

I believe you'll need to also use the --include flag in your command (see docs for reference)

aws s3 sync --include "*" --exclude "upload/*" --exclude "upload_qa/*" cdn/ s3://cdn-accountname/qa/sitename

Be careful the order

When there are multiple filters, the rule is the filters that appear later in the command take precedence over filters that appear earlier in the command

Tested this out and working fine

home [~]: aws s3 ls s3://abc-fe-testing1
                           PRE folder1/
                           PRE folder2/
                           PRE folder3/
                           PRE folder4/
home [~]: aws s3 ls s3://abc-fe-testing2
home [~]: aws s3 sync s3://abc-fe-testing1 s3://abc-fe-testing2 --exclude "folder1*" --exclude "folder2*"
copy: s3://abc-fe-testing1/folder4/file_from_folder4.txt to s3://abc-fe-testing2/folder4/file_from_folder4.txt
copy: s3://abc-fe-testing1/folder3/file_from_folder3.txt to s3://abc-fe-testing2/folder3/file_from_folder3.txt
home [~]: aws s3 ls s3://abc-fe-testing2
                           PRE folder3/
                           PRE folder4/
maafk
  • 6,176
  • 5
  • 35
  • 58
  • Still, it is not working. I even included the exclude folders in single quotes and also tried several orders but still, both the upload folders are synced to s3. I even gone through aws cli page and tried other options as well but none of them worked. Its there any other things I could try – MichealMills Feb 13 '18 at 11:27
  • Working okay for me. I updated the answer to show how it worked for me – maafk Feb 13 '18 at 15:22
  • 1
    I have used below commands in the Jenkins execute shell section, but still, the excluded folders are synced to s3 `aws s3 sync cdn/ s3://cdn-accountname /qa/sitename --exclude "uploads*" --exclude "uploads_qa*" aws s3 sync cdn/ s3://cdn-accountname /qa/sitename --exclude "cdn/uploads*" --exclude "cdn/uploads_qa*"` here cdn is a folder within my bitbucket repository and this folder has various directories among which I need to exclude "upload" and "upload_qa" directories while sync to s3 – MichealMills Feb 13 '18 at 17:10