3

Scenario:

I have been working on implementing a concourse ci pipeline for over a month now and my single yml file has grown quiet a bit. I understand that it is best practice to breakup the pipeline into several files and reference them in your pipeline.

Question:

Can someone please provide what the best practice is to structuring your concourse ci pipeline?

My thought process:

offering-pipeline | |_ ci: | | | |_ images: | | |_ Dockerfile | | | |_ misc: | | |_ python-requirements.txt | | | |_ ci-pipeline.yml | |_ project: |_ project-pipeline.yml | |_ jobs | |_ scripts : | |_ build: | |_ build_xyz. | |_ deploy: | |_ deploy_xyz. | |_ test: | |_ test_xyz. | |_ publish: |_ publish_xyz.

Thanks,

-Abe.

Abraham
  • 230
  • 3
  • 15

1 Answers1

2

A First step would be to extract all tasks to files. I have a tasks folder, a templates folder, and a script folder for each pipeline. Above these (in the root) I have a pipeline.yml containing the pipeline root structure, and a Makefile and Makefile.constants for setting up the pipeline in concourse..

Since I dont have that many build,test,publish tasks I have a naming convention on them instead of loads of folders with 1-4 files in each.

The tree inside my pipeline folder in atom:

enter image description here

Note: the pipeline.yml file is still pretty long (~500 lines)

The Makefile, the ${} comes from the included contant-files:

#Setup Makefile constants for urls etc:
include ../Makefile.constants

#Setup Makefile constants for  repo:
include ./Makefile.constants


set-pipe:
    fly -t dev_refactor set-pipeline \
        --config pipeline.yml \
        --pipeline ${PIPELINE} \
        --var "client-repo=${CLIENT_REPO_URI}" \
        --var "client-branch=${CLIENT_BRANCH}" \
        --var "server-repo=${SERVER_REPO_URI}" \
        --var "server-branch=${SERVER_BRANCH}" \
        --var "private-key=$$(cat ~/.ssh/id_rsa_no_passphrase)" \
        --var "docker-registry=${DOCKER_REGISTRY}" \
        --var "docker-registry-cert=$$(cat ../keys/docker-registry/docker-registry.crt)" \
        --var "docker-registry-server-dist=${DOCKER_REGISTRY}/server" \
        --var "docker-registry-client-dist=${DOCKER_REGISTRY}/client" \
        --var "docker-registry-node=${DOCKER_REGISTRY}/node" \
        --var "docker-registry-maven=${DOCKER_REGISTRY}/maven" \
        --var "docker-registry-protractor=${DOCKER_REGISTRY}/protractor" \
        --var "docker-registry-npm-cache=${DOCKER_REGISTRY}/npm-cache" \
        --var "docker-registry-soap=${DOCKER_REGISTRY}/soap-ui" \
        --var "reports-server=${REPORTS_SERVER}" \
        --var "concourse-url=${CONCOURSE_URL}" \
        --var "nexus-url=${NEXUS_URL}"

.PHONY: set-pipe

destroy-pipe:
# Destroy the pipeline in concourse
    fly -t dev_refactor destroy-pipeline \
        --pipeline ${PIPELINE}
.PHONY: destroy-pipe
David Karlsson
  • 9,396
  • 9
  • 58
  • 103
  • Yeah I'm at ~330 lines right now for my pipeline, what exactly does your Makefile, Makefile.constants do? – Abraham Jan 09 '17 at 17:07
  • Hey David, thanks for your reply and things are making a bit more sense now. I see that your Makefile.constants has the constant definitions and you import them into your Makefile. I like that idea, follow up question and kinda off topic what made you use Makefiles and are there any alternatives? – Abraham Jan 10 '17 at 15:43
  • Sorry I kinda forgot to ask what do you have in your templates folder? – Abraham Jan 10 '17 at 15:44
  • Some docker files and stuff that we use for setting up an integration test stack for example. you could use an ordinary bash script as a replacement for Makefiles.. – David Karlsson Jan 10 '17 at 19:26
  • Hey @david thanks for replying, I'm just full of questions. Who might be the consumer of the makefile in your repo? – Abraham Jan 11 '17 at 13:54
  • With consumer you mean: user? use it as a template for setting the initial pipeline and after changes for updating a pipeline... In my case (A bigger setting) I guess a platform team could create an initial boilerplate and push it to the different teams who will use the build server and then have them maintain it with some guidance.. – David Karlsson Jan 12 '17 at 08:42