In GCP, through Cloudbuild how can I ensure that only those steps get triggered for which changes have happened in file

Question

My issue is that, given the below yaml file, if I'm making changes for example in any file of "dir: process/cbd-bu-data", Cloud Build runs all the steps serially when triggered. This leads to wastage of time.

I want that only that step runs in cloudbuild for which the changes have been made in file of that directory. What should I do to achieve this?

Here's my cloudbuild.yaml file:

steps: 
  - args: 
      - beta
      - functions
      - deploy
      - "--runtime=python37"
      - "--trigger-http"
      - "--entry-point=process_cbd_group_data"
      - process_cbd_group_data
      - "--region=us-central1"
    dir: process/cbd-group-data
    name: gcr.io/cloud-builders/gcloud
  - args: 
      - beta
      - functions
      - deploy
      - "--runtime=python37"
      - "--trigger-http"
      - "--entry-point=process_cbd_bu_data"
      - process_cbd_bu_data
      - "--region=us-central1"
    dir: process/cbd-bu-data
    name: gcr.io/cloud-builders/gcloud
  - args: 
      - beta
      - functions
      - deploy
      - "--runtime=python37"
      - "--trigger-http"
      - "--entry-point=process_cbd_structure_data"
      - process_cbd_structure_data
      - "--region=us-central1"
    dir: process/cbd-structure-data
    name: gcr.io/cloud-builders/gcloud

Where are these files located? What is the source of the trigger? A bit more context on the trigger would be better. Although seems like sharding the triggers would be the best approach here. — bhito, Apr 04 '20 at 09:43
The location of file is present in tag "dir" for every step.Trigger is set for cloud repo any branch changes. — SUNDARAM SRIVASTAVA, Apr 05 '20 at 00:52
With your use case then the best approach would be having different triggers (3 in your use case) that listen to different tags or branches, being each of these specific for the file changes that you want to listen to. At the moment making Cloud Build steps execute when a certain file changes is not available. — bhito, Apr 05 '20 at 08:28
Hi, @bhito, could you add your comment as an answer so the community can benefit from it by it having more visibility? — asbovelw, Apr 07 '20 at 13:42

score 9 · Answer 1 · answered Jun 29 '20 at 12:21

You cannot do this from one cloudbuild. What you can do is create three different build triggers with the --included-files option. I think it is not convenient to accomplish the same thing with branches or tags as I read in the other answer. Read the documentation for more details.

Your git repository layout:

function_one/
   main.py
   cloudbuild.yaml

function_two/
   main.py
   cloudbuild.yaml

function_three/
   main.py
   cloudbuild.yaml

cloudbuild.yaml

Layout of the parent cloudbuild.yaml:

steps:
  - name: 'gcr.io/cloud-builders/gcloud'
    entrypoint: 'bash'
    args:
      - '-c'
      - |
        cloud beta builds triggers create github build_one --included-files "function_one/*" --repo-name=XXX --repo-owner=XXX --branch-pattern=$BRANCH_NAME
        cloud beta builds triggers create github build_two --included-files "function_two/*" --repo-name=XXX --repo-owner=XXX --branch-pattern=$BRANCH_NAME
        cloud beta builds triggers create github build_three --included-files "function_three/*" --repo-name=XXX --repo-owner=XXX --branch-pattern=$BRANCH_NAME

Layout of the sub cloudbuild.yaml:

steps: 
  - args: 
      - functions
      - deploy
      - "--runtime=python37"
      - "--trigger-http"
      - "--entry-point=process_cbd_group_data"
      - process_cbd_group_data
      - "--region=us-central1"
    name: gcr.io/cloud-builders/gcloud

score 2 · Accepted Answer · answered Apr 08 '20 at 07:04

With your use case then the best approach would be having different triggers (3 in your use case) that listen to different tags or branches, being each of these specific for the file changes that you want to listen to. At the moment making Cloud Build steps execute when a certain file changes is not available.

score 0 · Answer 3 · answered Apr 24 '21 at 18:16

If you wanna do it with gcloud CLI

gcloud beta builds triggers create cloud-source-repositories \
--repo=REPO_NAME \
--branch-pattern=BRANCH_PATTERN \ # or --tag-pattern=TAG_PATTERN
--build-config=BUILD_CONFIG_FILE \
--substitutions=_VARIABLE="VALUE"\ 
--included-files "DIRECTORY_NAME/**"

Note:-
--included-files "directory_name/**" will detect all directories and files recursively.
--included-files "directory_name/*" will only look for files in that particular directory.

Example:-

gcloud beta builds triggers create cloud-source-repositories \
--repo=test-repo \
--branch-pattern=master \
--build-config=workflows/cloudbuild.yaml \
--substitutions=_REGION="asia-southeast1"\
--included-files "src/**"

In GCP, through Cloudbuild how can I ensure that only those steps get triggered for which changes have happened in file

3 Answers3

Linked