How to parameterise concourse task files

Question

I'm pretty impressed by the power and simplicity of Concourse. Since my pipelines keep growing I decided to move the tasks to separate files. One of the tasks use a custom Docker image from our own private registry. So, in that task file I have:

image_resource:
  type: docker-image
  source:
    repository: docker.mycomp.com:443/app-builder
    tag: latest
    username: {{dckr-user}}
    password: {{dckr-pass}}

When I do a set-pipeline, I pass the --load-from-vars argument to load credentials etc from a seperate file.

Now here's my problem: I notice that the vars in my pipeline files are replaced with the actual correct values, but once the task runs, the afore mentioned {{dckr-user}} and {{dckr-pass}} are not replaced.

How do I achieve this?

check https://concourseci.slack.com/archives/general/p1488384207016956 for the discussion about this topic on Slack — Michael Lihs, Mar 14 '17 at 12:39
This is soon to come. It has been merged into `Master` on the Concourse GitHub repo. See > https://github.com/concourse/concourse/issues/454 ... — Lars Bingchong, Jan 08 '19 at 10:20

score 6 · Answer 1 · edited May 23 '17 at 12:34

6

In addition to what was provided in this answer

If specifically you are looking to use private images in a task, you can do the following in your pipeline.yml:

resources:
- name: some-private-image
  type: docker
  params:
    repository: ...
    username: {{my-username}}
    password: {{my-password}}

jobs:
- name: foo
  plan:
  - get: some-private-image
  - task: some-task
    image: some-private-image

Because this is your pipeline, you can use --load-vars-from, which will first get your image as a resource and then use it for the subsequent task.

You can also see this article on pre-fetching ruby gems in test containers on Concourse

The only downside to this is you cannot use this technique when running a fly execute.

edited May 23 '17 at 12:34

Community

1
1

answered Mar 14 '17 at 17:41

materialdesigner

1,492
10
13

the problem that I have with that solution is that the pipeline needs to know, which image a task requires - I would rather see that in the configuration of the task, not in the pipeline. This is not possible if you need any parameters in the task.yml and that is a flaw in the design of the pipeline / task concept in my eyes. It breaks the possibility of having tasks in a generic library, hence avoids re-use of tasks. – Michael Lihs Mar 17 '17 at 13:37

sercanturkmen · Answer 2 · 2018-09-21T10:24:50.313

As of concourse v3.3.0, you can set up Credential Management in order to use variables from one of the supported credential managers which are currently Vault, Credhub, Amazon SSM, and Amazon Secrets Manager. So you don't have to separate your task files partially in the pipeline.yml anymore. The values you set in the Vault will be also accessible from the task.yml files.

And since v3.2.0 {{foo}} is deprecated in favor of ((foo)).

Using the Credential Manager you can parameterize:

source under resources in a pipeline
source under resource_types in a pipeline
webhook_token under resources in a pipeline
image_resource.source under image_resource in a task config
params in a pipeline
params in a task config

For setting up vault with concourse you can refer to:

https://concourse-ci.org/creds.html

score 1 · Answer 3 · answered Mar 14 '17 at 17:35

You can always define tasks in a pipeline.yml... For example:

jobs:
- name: dotpersecond
  plan:
  - task: dotpersecond
    config:
      image_resource:
        type: docker-image
        source:
          repository: docker.mycomp.com:443/app-builder
          tag: latest
          username: {{dckr-user}}
          password: {{dckr-pass}}
      run:
        path: sh
        args:
          - "-c"
          - |
            for i in `seq 1000`; do echo hi; sleep 2; done

How to parameterise concourse task files

3 Answers3