31

I'd like to automate deploying our site to AWS S3. I've written a node script to automate building and uploading the site, but I'd like to have the script automatically run whenever the master branch of our repo is updated on github.

I looked into AWS CodeDeploy, but it looks like that's for specifically deploying to EC2. I've also looked at AWS Lambda, but there doesn't seem to be a clear way to pull a copy of the repo using git so I can run the script.

Any services (preferably tied to AWS) that I can use?

Matthias
  • 143
  • 1
  • 8
Mike Douglas
  • 3,345
  • 2
  • 28
  • 30

8 Answers8

12

I had the same goal some time ago and have now released a little tool, which solves the problem at least for me. It uses AWS Lambda and deploys a specific branch of the repository to S3 after push. You can take full advantage of the GitHub deploy key, which has fewer permissions as personal access tokens and can be configured per repository.

Please take a look at github-bucket, it might help you too.

github-s3-deploy-architecture

Matthias
  • 143
  • 1
  • 8
10

Rather than using an AWS service directly (as you say they nearly all expect a much more complicated setup, deploying to EC2 etc), you might be better off using a CI provider such as Shippable, Codeship or Wercker.

These all have the ability to fire from git updates, run build commands, install utilities into their CI images/containers and copy files to S3.

There's probably some startup which has built an exact tool for your purpose, but they haven't appeared on my radar yet :-)

ocean
  • 1,335
  • 15
  • 26
9

I know it's not git deploy.... But Instead of setting up a CI box, I just used s3cmd.

http://s3tools.org/s3cmd

Executing this command syncs my build directory with s3.

s3cmd sync -r ~/code/mysite/build s3://www.mysite.com --delete-removed

I'm using it on Linux. Not sure what their OSX and Windows stories are.

If you're really after a git push solution, you could set up a timed job which pulls your git repo to a folder and then executes this against it. I do this elsewhere on a cheap Linux VM. Unless you're going to go full CI though, there's probably not much point.

Damien Sawyer
  • 5,323
  • 3
  • 44
  • 56
  • This could get added into a git post-commit hook and it would do. – Zlatko Apr 13 '16 at 08:36
  • Yeah, good idea. It just occurred to me, with the recent announcement that Windows will be able to run Linux binaries, s3cmd will probably work on Windows as well! So cool. https://blogs.windows.com/buildingapps/2016/03/30/run-bash-on-ubuntu-on-windows/ – Damien Sawyer Apr 13 '16 at 21:22
7

You can set this up with a very simple 2-step CodePipeline. Basically, you just fill out the different sections in the AWS Console. There's no need for a separate CI tool and its added complexity.

In the first step of the pipline, pull from Github and store in S3. You can easily set this up through the AWS Console.

In the next CodeDeploy step, you can use the AWS CLI (pre-installed in CodeDeploy) to do a

cd /path/to/public/directory && aws s3 sync --acl public-read --delete . s3://your.bucket.name

You'll have to set the environment variables for AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY so that AWS CLI can run during your deploy step, and that can also be done in the AWS console for CodeDeploy, in the Advanced section under "environment variables". Once the environment variables have been set and if that AWS user has the correct permissions, you can run any aws-cli command you want inside your CodeDeploy.

Once this is done, when you check in to Github, CodePipeline will kick off, and a few minutes later your files will be on S3.

Patrick Chu
  • 1,513
  • 14
  • 15
  • How do you set up the pull from Github to S3? Is that with Lambda? Or the CodePipeline? – LC1983 Dec 03 '17 at 00:01
  • In your CodePipeline wizard, it asks you for the source. Github is one of the supported sources, so choose Github as your source. You'll need to set up Github authentication so that your CodePipeline has permissions to read your Github account (if it's a private repo), but that's all done through the console. The first time I did this, I had no problems doing everything through the console -- the steps are pretty clear. – Patrick Chu Dec 27 '17 at 20:01
  • I can get the code in, but can’t set S3 as n output. – LC1983 Dec 27 '17 at 20:02
  • 1
    You have to write the build scripts to run on CodeDeploy. If you just want to copy the files to S3, your build script would be just the single line of script shown in my answer. – Patrick Chu Jan 21 '18 at 16:44
  • Thanks, that sounds interesting - I had started writing something from scratch in Lambda, but CodePipeline/CodeDeploy sounds like a better idea. Can you link me to a tutorial for how to set this up, as I can't see how/where to put in the line of code you suggested. Nor where I would specify the environment variables. – LC1983 Jan 21 '18 at 21:47
  • 1
    Hi. Actuallly you don't even need a pipeline to do that copy. The only thing you need is a Build Project, that should be created in CodeDeploy. You can link your GitHub account since there and specify in addittional conf. that you want run the build each time your repository changes. It is necessary to create the 2 env. variables that @PatrickChu said and in the BuildSpec section you should select "Use a build command". In my case I used this command: `aws s3 sync --acl public-read --delete . s3://MY-BUCKET-NAME-HERE` I previously create a new Bucket in S3 where my data is being placed. – Ruben Nov 11 '18 at 21:51
  • I would like to mention, that you will need to grant AWS access to all of your repositories of your GitHub account for this solution. If you are okay with that, the solution is absolutely great and preferable as it requires no third party software or CI system. I have not checked, which content type will be set for those uploaded files, but I would guess, that content type detection might not be as great as expected for special file types. – Matthias Nov 27 '18 at 10:34
6

If you are using TravisCI, the deployment is pretty straight forward. The only part you will need to add to your .travis.yml is :

deploy:                         
  provider: s3                         
  access_key_id: "YOUR AWS ACCESS KEY"
  secret_access_key:
    secure: "w/DlbHt1+IAMENCRYPTED"
  bucket: "YOUR BUCKET"

My blog post explains all the details for the AWS side (user setup, IAM and S3 bucket configurations) as well as the github and travisCI side.

Michal Frystacky
  • 1,418
  • 21
  • 38
1

Perhaps overpowered for your simple use case, but you could create a very simple CodePipeline to push your github repository to S3.

Dave Maple
  • 8,102
  • 4
  • 45
  • 64
0

i also recommend to use codeship,simple and easy, but you need to create IAM user with proper permission (which is policy) to S3 bucket.

the basic plan for codeship is free.

Well there might be a problem so far i can see codeship will not remove files as you remove files in github, after all, s3 is not github repo, but anyway, the putObject operations for lots of github update just works good enough to me.

Gabriel Wu
  • 1,938
  • 18
  • 30
  • Not proved this works, but read that: if you use _aws s3 sync_ _--delete (boolean) Files that exist in the destination but not in the source*are deleted during sync_ From: http://docs.aws.amazon.com/cli/latest/reference/s3/sync.html – Nigel Jul 05 '17 at 14:49
0

I have used Deploy Bot in the past and I have been quite happy with it.

It can push to S3 or even FTP via Git and also it can run your build scripts and even push a notification to Slack for you.

https://deploybot.com/

Anthony Delgado
  • 87
  • 1
  • 1
  • 7