0

I'm trying to set up a build pipeline for a web service in AWS. The plan is to have the service run in an Auto Scale Group, and use Jenkins to build a one off EC2 instance, run tests and, on success, image the instance and pass it on to the existing Auto Scale Group.

To create all the necessary resources for this, I've created two CloudFormation templates, one to build the Autoscale Group and surrounding resources, and one to build a one off testable instance.

I've noticed a problem with this though: There are different base images for each region and each EC2 Instance Type. This means, Jenkins needs to know the region and Instance Type to build for.

The ideal solution would be a way of selecting the target CloudFormation stack, and pulling the information (region, instance type, etc) from there. This means if we make any changes to the Auto Scale Group, they would automatically be reflected in the Jenkins build. If we created a second Group, we can copy the Jenkins job and change one parameter to point it at the new Stack. But this doesn't seem to be an option...

These are the potential solutions I can think of but I don't particularly like any of them:

  • Hard code this information in both templates
    This goes against the idea of having CF templates, since I'd ideally like to keep these things flexible.

  • Send the information to an API, then have Jenkins look it up
    This is quite a lot of extra work as I have to build and maintain a machine just to store a couple of variables. Not to mention, I'm not sure I can output the information to an API from the script, so would have to wrap it in another script to be take the information from the AWS CLI tool.

  • Categorise the final image
    I could allow flexability in all parameters and then categorized the finalised image by those parameters. Then I can make sure the Auto Scale Group only loads images with the correct parameters. This would prevent using an incorrect base image but could result in building with the wrong information (if we change one we must remember to change the other).

It feels like what I want to do shouldn't be that hard but I can't work out the best way to do it.

DanielM
  • 147
  • 1
  • 8

3 Answers3

1

CloudFormation includes a pseudo-parameter AWS::Region where it can query the region in which it is running. See http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/pseudo-parameter-reference.html

You can use mappings to have a list of AMI images where you select your AMI based on the region. See http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-findinmap.html

Usually, the choice of AMI is not affected by the instance type. However, I'm not familiar with your use-case, so it may matter.

As for your workflow, I would not "make changes to" your Autoscaling group. It should go the other way: make changes to your CloudFormation templates, which in-turn update your stacks to update your Autoscaling groups.

So you would:

  1. Build your single instance from your CFN template, confirm it works and build your new AMI.
  2. Take the new AMI image ID and update your Autoscale group's CFN template.
  3. Update your Autoscale group by updating it's stack

The danger of making changes directly to your autoscaling group is that those changes won't be reflected in the stack's template. If you had to rebuild the stack or make a copy, it'll be missing those changes. Ideally, resources created from a CloudFormation stack should be treated as read-only and only updatable by making changes to the CFN template.

Matt Houser
  • 10,053
  • 1
  • 28
  • 28
0

A slightly better option that the ones listed above (and actually a possibility) is to use the AWS CLI tool to get information about the current stacks. This could be wrapped in a script to take the returned JSON and find the desired parameters / ouputs.

It still obfuscates it a little by moving it to an external script rather than having it in the CloudFormation Template so I'm going to leave the question open in the hopes someone has a better answer.

Here's an absurdly simple example:

<?php

$result = `aws cloudformation describe-stacks --stack-name=GROUPSTACKNAME`;
$stack = json_decode($result);
$stack = $stack->Stacks[0];

if(!$stack) {
    throw new Exception("Stack no found");
}

$parameters = [];
foreach($stack->Parameters as $param) {
    $parameters[$param->ParameterValue] = $param->ParameterKey;
}

echo http_build_query($parameters);
DanielM
  • 147
  • 1
  • 8
0

Define the information you need to transfer as Outputs from the first stack. Use pseudo parameters for the region, etc.

When you create the second stack, call describe-stack on the first one to get the Outputs, and the pass the values on as parameters to the second stack.

bsvingen
  • 126
  • 3