2

I have a Cloud Build trigger that moves files from my Cloud Source Repo to Cloud Composer GCS Bucker.

I'm using the following YAML file as a starting point.

steps:
- name: ubuntu
  args: ['bash', '-c', "echo '$COMMIT_SHA' > REVISION.txt"]
- name: gcr.io/cloud-builders/gsutil
  args:
    - '-m'
    - 'rsync'
    - '-d'
    - '-r'
    - ''
    - 'gs://${_GCS_BUCKET}/dags'

However, my Git structure looks like this:

.
|   project_strcuture.png
|   log.txt
|   README.md
|   cloudbuild.yaml
|   requirments.txt
|              
+---PROJECT_A
|       Project_A_PROD.py
|       Project_A - flow chart.png
|       README.md
|                      
+---Project_B_APAC
|   +---Project_B1_APAC
|   |       Project_B1_APAC_flow_chart.png
|   |       Project_B1_APAC.py
|   |       README.md
|   |       
|   +---Project_B2_APAC
|   |       Project_B2_APAC.py
|   |       Project_B2_APAC_flow_chart.png
|   |       README.md
|   |       
|   \---Project_B3_APAC
|           Project_B3_APAC.py
|           Project_B3_APAC_flow_chart.png
|           README.md
|
\---Global_Pipeline
|   \---API_BQ_Data_Validation
|   |       compare.csv
|   |       GlobalBQ_comaprison.py
|   |       GlobalBQ_comaprison.ipynb

I want only specific python files (valid Cloud Composer DAGs) to move from my Git to Cloud Composer. Is there any way to do it?

Abhinav Dhiman
  • 745
  • 3
  • 17
  • You can exclude the files which you don't want to push using .cloudignore. Please check the [following documentation if it helps you.](https://cloud.google.com/sdk/gcloud/reference/topic/gcloudignore) – Rajeev Tirumalasetty Nov 12 '21 at 21:41
  • I changed my approach. Instead of filtering files while ingestion, I am filtering files using a Python script after ingestion to the bucket. – Abhinav Dhiman Nov 18 '21 at 05:40
  • If you managed to solve the issue, then can you please post the answer below? Did you give a try to use `.cloudignore`? – Rajeev Tirumalasetty Nov 25 '21 at 13:12

0 Answers0