Automation of releases of microservices-based application

Question

We are working on the application that consists of many standalone services. It has advantages over the single monolithic application, but not when we do releases.

We do weekly release cycles. Each service/component located in the separate git repository. 'A release' - is several features that we put into wild. Usually only several components should be updated. We manage servers using saltstack. To make a release salt scripts update component's versions using git.latest state. The problem is to specify right versions.

This is where the manual work that I'd like to automate. To update versions I have to manually check each component's repository, merge development branch into master and tag according to symantec versioning rules. Then I write new version in salt scripts. We have over 10 components so this is rather boring and error prone process.

Probably we doing it wrong, I'll be glad to hear any advices how to do it better, thanks.

Answer 1

Firstly, I'd suggest following a convention for the release tags of your components. In the simplest case that just would be the freshest git tag on each of the repositories.

Then, you could create a mapping script (say, it's called map_versions) enumerating the release (latest) git tags for all the repositories, and storing that mapping somewhere for the SaltStack to pick it up -- to be used as the revision-s in the git.latest states.

The same mapping script can be also used to prepare the develop or master branches of all the components for deployment, -- all the revision values will be switched to develop or master.

Thus, your workflow will be:

// In the dev environment:

$ map_versions develop
$ salt * state.highstate

// Do the development, until all the stable features
// are merged back into master. Then, in production:

$ map_versions master
$ salt * state.highstate

// Make sure everything works fine; then, manually tag
// the new release versions for all the repos.

$ map_versions tags
$ salt * state.highstate

After which, all the released components in production are tagged.

You can also save some time with automatic git-tagging script for all your deployable components. The script would check if anything has changed in the master since the last tag, and, if it has, it would stick a new git tag on the repo; say, just the today's YYYY-MM-DD. Then, those tags would be picked up by the map_versions tags.

Answer 2

You could keep explicit version mapping for every component you want to include in the release (and possibly other metadata info as needed) in a separate git repo which becomes your master SCM control knob. This offers several advantages:

• not mixing scripts/code with metadata info (which is error prone)
• you can code your scripts to simply handle the versioning info from this master git repo, no need to change scripts for every release
• you only need to track/tag the master git repo since it contains all the metadata info about all the other components needed in the release - less SCM churn
• you can quickly access relevant metadata info for all components via this single tiny repo, no need to pull the entire set of components (unless you also need to acccess their content specifically)
• you prevent pollution of the components' SCM logs with your particular release info (especially important if these comps are shared with other completely unrelated or 3rd party products which couldn't care less about your particular release cycle).

This doesn't eliminate the release steps you have to do, it just adds some order and can help with the automation.

Answer 3

I think the tool you're looking for is a git hook.

Personally I'd probably set up a server side post-receive hook [0] in your repository which takes the semantic tag and either automatically updates the software version in the pillar data, or creates a Salt event which triggers either an update or a deploy using the provided data.

There's also the option of an external pillar data source [1], where it can automatically get the most recent tag or commit on git's master branch.

In either case, I'd keep the git merge and tag a manual step.

[0] http://www.git-scm.com/book/en/v2/Customizing-Git-Git-Hooks

[1] http://docs.saltstack.com/en/latest/topics/development/external_pillars.html

Answer 4

After working for more than a year developing and managing releases of platform built of microservices I figured repeatable process that can be automated. More on this below.

Let's split release process into 3 phases:

1. understanding what should go out in release
2. preparing changes
3. pushing them in the wild

We're using Git and A successful Git branching model, which is rather questionable, I prefer FeatureBranch workflow, but that's a different story.

First phase: Understanding of what should go out

In our issue tracking tool, stories that should go out are marked as "Ready for Merge" (we use jira.com but it doesn't matter). I grab list of stories, run simple script that look like this mia review --cards=MIA-1022 MIA-988 MIA-1097 MIA-688. The output is a list of services affected by these stories, so that you don't need to go and manually review every story to see services affected, example output looks like this:

[+] [2/16] user-service: MIA-1198, MIA-2023
[+] [6/16] checkout-service:  MIA-1097 MIA-688
[+] [7/16] inventory-service: MIA-1022 MIA-988, MIA-1198, MIA-2023

Second phase: Preparing changes

Semi-manual process for me, because in some cases the "in-progress" stories from develop branch need to be ignored and can't go to master. But in most cases I can merge develop straight to master, and when I can, I have another command: mia merge --services=user checkout inventory. This command goes over specified services and creates Pull Requests to merge develop branch to master and prints out links to pull requests.

Third phase: pushing changes in the wild

To push something to staging environment and then to production, service has to have a version. Empirically we figured that if you do semver for services, and moreover if you do it only for services that have changes it'll be hard to understand the "latest". Because what if pace of development of checkout service significantly higher then inventory service, you end up with something like v3.3.6 in checkout and v1.2.0 in inventory.

So to solve this: we're tagging all services with same tag version composed of the year, month, day and rc version. Example: r2015052601, and then we also have a command mia diff r2015052401 r2015052601, which searches for specified tag in every service and prints a diff of changes between 2 tags. Part of me thinks that tagging all services with same version violates one of the principles of microservices architecture, but for us right now it solves major pain point of tags compatibility and understanding what's latest, because you can assume that latest tag exists everywhere, and if there were no changes, then there were no changes.

Thanks