Update ECS Task definition for background service from Codepipeline - without a load balancer

Question

Introduction

I am deploying a Django app using Celery for background tasks on Amazon ECS, and we're using CodePipeline for CI/CD. I would like to be able to split this up into three ECS Services, each running only one task - this is so they can be scaled independently. It is proving hard to do while still meeting two key design goals:

1. Continuous delivery of changes - must be automated
2. Infrastructure changes must be managed as code

So, fundamentally, updates to ECS Task Definitions need to be versioned in git and updated as part of the automated release process and when they change, the services using them need to be updated.

For the service that accepts the traffic, this all works fine. The issue is with those services on ECS that are performing background tasks. There, I'm hitting a roadblock in that:

• CodeDeploy Deployment Groups insist on being associated with a Load Balancer, and
• Any Deployment provider that deals with updating the Task Definition requires a Deployment Group.
  • I think this is limited to the "CodeDeploy" and "ECS Blue/Green" providers.
• Neither my "scheduler" or "worker" service accept traffic

So, it comes down to this: what kind of deployment can I do that doesn't require a load balancer, but will still allow me to update the task definition as part of the deployment?

Details

Now, to give you more specifics, the list of service I want is:

• "web" service - runs Django, exposed to ALB on port 8000
• "scheduler" service - runs Celery "beat", no exposed port
• "worker" service - runs Celery worker, no exposed port

For the "web" service, CI/CD is straightforward, we have a CodeDeploy Application, with a Deployment Group that is associated with the Application Load Balancer and has the correct target groups and this does a "Blue/Green" deployment.

We have built some custom tooling that generates a replacement taskdef.json and appspec.yml for each of the services. These tools are invoked during the Build phase of our pipeline and (for the "web" service) applied at deployment time; this is so that updates to the application environments and resources are also managed in code.

So the flow goes:

1. Build new docker container
2. Generate new taskdef.json from source templates - filling in resource IDs (secrets etc.) by querying the CloudFormation stack
3. Generate new appspec.yml with the revision number of the task definition incremented by 1
4. CodeDeploy creates a new revision of the application based on the new AppSpec and TaskDef (Build artifacts from previous step) and deploys the updated service on the cluster.

This works well for the "web" service, and I would like something similar for the other two services, but I cannot find a way to either: not have a Deployment Group, but still update the Task Definition; or have a Deployment Group but not have a Load Balancer (because there's no traffic to load balance).

Is there a trick to this? Or a deployment type I've missed that is aimed at background services?

I would appreciate any advice you have to offer. Thanks!

Answer 1

For posterity, the answer I came up with in the end was to create a lambda dedicated to re-deploying the ECS Services for celery beat and the workers. Then have CodePipeline deploy the Web service using a Blue/Green deployment and then call the lambda twice (in parallel): once for the scheduler service, once for the worker service.

None of the built-in deployment types were of any help at all in getting this going.