Blue/Green deployments with Auto Scaling Groups, CloudFormation and CodeDeploy

Question

I have tried setting up a Blue/Green deployment by copying AutoScalingGroup, however this leaves the CloudFormation stack detached from its original resources as CodeDeploy creates a new copy and deletes the original. I understand from another post (https://forums.aws.amazon.com/thread.jspa?messageID=861085) that AWS are developing improvements for this, however for now I am trying the following workaround. Any ideas would be really helpful.

CloudFormation creates the following:

• Elastic Load Balancer
• Target Group
• AutoScalingGroup One (with LaunchConfiguration)
• AutoScalingGroup Two (same as one but has no instances)
• DeploymentGroup (with In-Place DeploymentStyle) which deploys a revision to AutoScalingGroup One

After CloudFormation finishes, I do the following manually in the console:

1. I update the created Deployment Group to be of Deployment Style Blue/Green and set its original environment to be AutoScalingGroup One.
2. I add an instance to AutoScalingGroup Two
3. I create a deployment in CodeDeploy. However, this does not work as when a new instance is attached to AutoScalingGroup Two, it gets added to the TargetGroup immediately and does not pass health checks.

Any ideas on how to implement a set of resources with CloudFormation that will make blue green deployments simple, i.e. one click in CodeDeploy and CloudFormation resources still remaining intact?

Answer 1

With regard to the initial issue you are describing, did you experiment with the Health Check Grace Period? That should prevent the problems you describe with the failing health check when the instance hits the target group.

An alternative approach (which has plenty of its own downsides) is to adapt the CloudFormation template to compensate for the behavior when CodeDeploy replaces the ASG in a Blue-Green deployment.

1. Within the ASG template, create a "yes/no" parameter called "ManageAutoScalingGroup". Create the ASG conditionally on the value of this parameter being "yes". Set a deletion policy on the ASG of retain so that CloudFormation will leave the group in place when the parameter is changed to "no".
2. Spin up the group with a default "yes" on this parameter.
3. Once the instances are healthy, and CodeDeploy has completed an initial in-place deployment, you can change the DeploymentGroup to use Blue-Green where CodeDeploy will replace your ASG.
4. Be sure to update the ASG and change ManageAutoScalingGroup to "no". CloudFormation will delete the reference from your stack, but it will leave the resource in place.

This will give you the one-click deployments you desire through CodeDeploy, but be aware that it comes with some costs:

• CodeDeploy will not copy the TargetGroup parameter of your Auto Scaling Group (as described by others in https://forums.aws.amazon.com/thread.jspa?threadID=249406&tstart=0). You should be able to work around this with a clever use of CloudWatch event rules and SSM Automation to mark the instance unhealthy when the ALB changes its status.
• The copies that CodeDeploy produces seem to be fairly unreliable. At least once, I've seen my LaunchTemplate version reset to an incorrect value. I've also run into scenarios where the deployment group lost track of which ASG it was supposed to track.
• Continuing to apply changes from your template to the ASG is a hassle. The process to "refresh" the group is: 1) Revert the parameter described earlier such that CloudFormation will generate a new group. 2) Modify the deployment group to target this group and complete an in-place deployment. 3) Modify the deployment group to restore Blue-Green deployments and update your stack accordingly.

I'm not too impressed with CodeDeploy in this department. I'd love to see them work in the same fashion as an ASG that is set to replace itself on application of a new LaunchTemplate version. If you are feeling a bit ambitious, you could mimic this behavior by leveraging Step Functions with ASG instance lifecycle hooks. This is a solution that I'm considering once I have the time.