10

I'm investigating whether or not CodePipeline will work for our use case:

We have several hundred repositories, all hosted with CodeCommit. The build/test/release process for all of these is identical, aside from minor configuration.

What I'd like to do is set up a pipeline that will build/test/release changes made in any of those repositories. I don't want to build all repositories, only the one that changed.

I'd rather not create 100 identical pipelines with different source repositories...

Is this possible? If not, are there any workarounds?

Tom
  • 843
  • 7
  • 12

2 Answers2

6

CodePipeline is designed around a model of one pipeline per project / service.

Trying to combine multiple independent release processes into a single pipeline is challenging due to the superseding behavior where newer changes can replace older changes while waiting for a stage to be come free to avoid unnecessary releases.

It will also make certain features such as the release history much less useful.

In your scenario, 100 pipelines is probably the best approach. CodePipeline recently raised the default limit of pipelines per account per region to 300 so you should have no problems in terms of limits.

You can use tools like CloudFormation to simplify setup and configuration of many similar pipelines. Also keep in mind that CodePipeline doesn't charge for pipelines which have no releases during a given month, so if you have many pipelines but you only actually make releases in a few of them per month then it won't cost you as much.

TimB
  • 1,457
  • 8
  • 10
  • This is the conclusion I've also come to. The service is quite clearly geared toward a single project/application, which is great but doesn't work for my use case. We could work around one or two things, but ultimately we'd be fighting against the service and causing ourselves more headaches in future. The current plan is to make use of CodeCommit, CodeBuild, Lambda, SQS, and ECS to essentially run our own custom pipeline. If that doesn't work out we'll go down the route of one pipeline per project, like you suggested. Cheers! – Tom Dec 21 '18 at 22:13
  • @TimB When you say code pipeline is designed around a model of one pipeline per project what do you mean, exactly? Say I have one project but it uses a poly\micro repo approach and I have some terraform in one repo, services in 10 repos and I have 11 code pipelines, one for each repo. How does AWS recommmend you coordinate a release across all 11 pipelines? Or is this something code pipeline is not really intended to solve? Take a simple 3 tier app, and I make a stored procedure change, a platform change, an API change and a UI change across 4 repos and need to deploy 4 pipelines at once? – berimbolo Mar 28 '21 at 09:03
1

You may have to use a mix of CloudWatch rules + AWS Lambda + CodePipeline

CloudWatch Rules

  1. CodeCommit State Changes --> CloudWatch Rules --> Invoke Lambda (create a file which contains information with regard which repo and branch etc (ex: state-change.json). -> upload file to S3 bucket (ex: pipeline_source).

  2. S3 (pipeline_source) Events -> CloudWatch --> Trigger pipeline

CodePipeline

[ Source (S3 state-change.json, BuildSpec.yml, etc..)] --> [ Build (CodeBuild, custom script reads through the statechange.json file to determine what to build and deploy] -->
[ Deploy (...)]

Approach # 2 Parallel execution (different repos updated at the same time )

I don't think it's possible to wire up a single pipeline to meet your needs because of how pipeline runs the revisions. An alternative is to replace codePipeline with an step function state machine. It allows parallel execution. You can wireup the step function with actions that fits your CI/CD needs. I had a quick look at aws service integrations not many are supported. You need to use lambda function to invoke other aws services for ex: codebuild

CodeCommit state changes --> CloudWatch Rule --> Invoke Step function state machine

mockora
  • 106
  • 2
  • Thanks, I had a similar idea. The problem is that CodePipeline will batch/queue revisions until the next stage is free. If a later revision arrives in the batch/queue, it is replaced. – Tom Dec 19 '18 at 06:46
  • From the docs: Because only one revision can run through a stage at a time, CodePipeline batches any revisions that have completed the previous stage until the next stage is available. If a more recent revision completes running through the stage, the batched revision is replaced by the most current revision. This means that if five repositories all push a revision at the same time, only the first and last revision will probably make it through. – Tom Dec 19 '18 at 06:54