Questions tagged [google-cloud-composer]

Google Cloud Composer is a fully managed workflow orchestration service, built on Apache Airflow, that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers.

Cloud Composer is a product of Google Cloud Platform (GCP). It is essentially "hosted/managed Apache Airflow."

The product allows you to create, schedule, and monitor jobs, each job being represented as a DAG (directed acyclic graph) of various operators. You can use Airflow's built-in operator definitions and/or or define your own in pure Python.

While technically you can do data processing directly within a task (instantiated operator), more often you will want a task to invoke some sort of processing in another system (which could be anything - a container, BigQuery, Spark, etc). Often you will then wait for that processing to complete using an Airflow sensor operator, possibly launch further dependent tasks, etc.

While Cloud Composer is managed, you can apply a variety of customizations, such as specifying which pip modules to install, hardware configurations, environment variables, etc. Cloud Composer allows overriding some but not all Airflow configuration settings.

Further technical details: Cloud Composer will create a Kubernetes cluster for each Airflow environment you create. This is where your tasks will be run, but you don't have to manage it. You will place your code within a specific bucket in Cloud Storage, and Cloud Composer will sync it from there.

1225 questions
4
votes
1 answer

Why does this code to get Airflow context get run on DAG import?

I have an Airflow DAG where I need to get the parameters the DAG was triggered with from the Airflow context. Previously, I had the code to get those parameters within a DAG step (I'm using the Taskflow API from Airflow 2) -- similar to this: from…
4
votes
1 answer

Importing custom plugins in Airflow 2 [Cloud Composer]

I have a directory structure as such: airflow_dags ├── dags │ └── hk │ └── hk_dag.py ├── plugins │ └── cse │ └── operators.py │ └── cse_to_bq.py └── test └── dags └── dag_test.py In the GCS…
juwa92
  • 43
  • 4
4
votes
3 answers

How to invoke a cloud function from google cloud composer?

For a requirement I want to call/invoke a cloud function from inside a cloud composer pipeline but I cant find much info on it, I tried using SimpleHTTP airflow operator but I get this error: [2021-09-10 10:35:46,649] {taskinstance.py:1503} ERROR -…
4
votes
2 answers

How to use Airflow Stable Rest API [Airflow version 2.0.0] deployed on GCP Cloud Composer

Env : We using GCP cloud composer to run airflow dags. Aim : To use apache airflow stable apis to trigger dags externally using REST. We tried to use airflow experimental apis to trigger dags externally by setting airflow override configuration in…
4
votes
1 answer

PythonVirtualenvOperator fails with TypeError: cannot pickle 'module' object

The following happens on Cloud Composer 2.1.xx I am trying to use the PythonVirtualenvOperator with templated parameters. Unfortunately the operator fails with the following error : TypeError: cannot pickle 'module' object Here's the code of my dag…
Will
  • 2,057
  • 1
  • 22
  • 34
4
votes
2 answers

What is the difference between GCP cloud composer and workflow?

The cloud workflow doesn't come with a scheduling feature. Apart from that, what are all the differences between these two services in terms of features? In which use case should we prefer the workflow over composer or vice versa?
4
votes
2 answers

How to set up dbt with Google Cloud Composer?

I am trying to install dbt on Google Cloud Composer but run into dependency issues. I have followed the instructions from this article: https://blog.doit-intl.com/setup-dbt-with-cloud-composer-ab702454e27b however at step 2: installing the packages…
4
votes
0 answers

Airflow 1.10.15. kills Zombie Tasks for seemingly no reason

We're using Cloud Composer (aka. Airflow) for a time now. Recently we updated to the newest version composer-1.16.5-airflow-1.10.15. Ever since we're experiencing some issues surrounding so called zombie tasks. Zombie tasks seem to arise if Airflow…
4
votes
1 answer

Running a Terraform command via Airflow operator

I'm running Apache Airflow on Cloud Composer (composer-1.14.2-airflow-1.10.14). I want to use Terraform to create infrastructure but I can't find any operators to do this. As a workaround I'm using BashOperator like this: create_vm=BashOperator( …
parakeet
  • 395
  • 1
  • 11
4
votes
1 answer

AttributeError: 'AuthorizedSession' object has no attribute 'configure_mtls_channel'

I was orchestrating two dataflow job with cloud composer and it was working fine for month. Suddenly the two jobs stopped working with the following error message: in download_blob…
4
votes
3 answers

Unable to trigger composer/airflow dag from Cloud function that triggers when there are changes in cloud storage

I have created and ran dags on a google-cloud-composer environment (dlkpipelinesv1 : composer-1.13.0-airflow-1.10.12). I am able to trigger these dags manually, and using the scheduler, but I am stuck when it comes to triggering them via…
4
votes
2 answers

Google Cloud Composer: Save on costs

I am trying to figure out how to save on costs via Google Cloud Composer. Is there anyway to spin down the server when none of your dags are running? Then spin it up again when a dag needs to run? It's costing way too much since I believe even…
Bigmoose70
  • 453
  • 6
  • 15
4
votes
2 answers

Google Cloud Composer DAG not appearing in UI

Was attempting to add a new DAG to our Google Cloud Composer instance - we have 32+ DAGs currently - and doing the usual things in https://cloud.google.com/composer/docs/how-to/using/managing-dags doesn't appear to be having any effect - we can't…
Yeager
  • 87
  • 7
4
votes
4 answers

Google Composer- How do I install Microsoft SQL Server ODBC drivers on environments

I am new to GCP and Airflow and am trying to run my python pipelines via a simple PYODBC connection via python 3. However, I believe I have found what I need to install on the machines [Microsoft…
4
votes
1 answer

Cloud Composer pricing

I've set up a cloud composer environment and I'm very confused with the pricing. In the pricing page, it's written that any environment would have at least 2 DB vCPU and 2 Web server vCPU. Further in the page, it's written that it's possible to…
Nicoowr
  • 770
  • 1
  • 10
  • 29