Questions tagged [google-cloud-composer]

Google Cloud Composer is a fully managed workflow orchestration service, built on Apache Airflow, that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers.

Cloud Composer is a product of Google Cloud Platform (GCP). It is essentially "hosted/managed Apache Airflow."

The product allows you to create, schedule, and monitor jobs, each job being represented as a DAG (directed acyclic graph) of various operators. You can use Airflow's built-in operator definitions and/or or define your own in pure Python.

While technically you can do data processing directly within a task (instantiated operator), more often you will want a task to invoke some sort of processing in another system (which could be anything - a container, BigQuery, Spark, etc). Often you will then wait for that processing to complete using an Airflow sensor operator, possibly launch further dependent tasks, etc.

While Cloud Composer is managed, you can apply a variety of customizations, such as specifying which pip modules to install, hardware configurations, environment variables, etc. Cloud Composer allows overriding some but not all Airflow configuration settings.

Further technical details: Cloud Composer will create a Kubernetes cluster for each Airflow environment you create. This is where your tasks will be run, but you don't have to manage it. You will place your code within a specific bucket in Cloud Storage, and Cloud Composer will sync it from there.

1225 questions
0
votes
1 answer

Is there a sensor for finding if a particular file type is present in a Google cloud storage?

I have a requirement in composer to poll a task till it finds a zip file in a bucket and run the subsequent tasks after the condition is met The best i could get is GoogleCloudStoragePrefixSensor. But this is with prefix and doesn't work for this…
0
votes
0 answers

create airflow dag dynamically looping though a dictionary

I am trying to add airflow dag dynamically looping through the dictionary keys and assigning keys as dag name. dags are creating fine but i am getting :"This DAG isn't available in the webserver DagBag object. It shows up in this list because the…
user8617180
  • 267
  • 6
  • 20
0
votes
2 answers

Running a code from an instance in Google Cloud Composer

I am new to google cloud composer. I have some code in google cloud compute engine - for eg: test.py Currently I am using Jenkins as my scheduler - and I'm running the code like below echo "cd /home/user/src/digital_platform…
0
votes
2 answers

Create Airflow connections on Cloud Composer using gcloud CLI

I am trying to create airflow connections on Cloud Composer using gcloud CLI. I follow the document and run the following comment. https://cloud.google.com/composer/docs/how-to/managing/connections#creating_new_airflow_connections gcloud composer…
Yohei Onishi
  • 1,362
  • 1
  • 19
  • 42
0
votes
2 answers

Airflow not recognising zip file DAG built with pytest fixture

We are using Google Composer (a managed Airflow service) with airflow v1.10 and Python 3.6.8. To deploy our DAGS, we are taking the Packaged DAG (https://airflow.apache.org/concepts.html?highlight=zip#packaged-dags) method. All is well when the…
krc
  • 21
  • 3
0
votes
2 answers

Airflow : pass parameter from python function to MySQL Operator

What I want to do : - compare missing ID from MySQL and BigQuery - get all data from MySQL where id in missing ID table = 'orders' def get_missing_ids(): aws_id = get_aws_id(table) bq_id = get_bq_id(table) missing_id =…
0
votes
1 answer

Error message while deploy a composer resource (GCP) with terraform

I am having an error with a terraform code, while deploy a GCP composer resource: google_composer_environment.composer-beta: googleapi: Error 400: Property key must be of the form section-name. The section may not contain opening square brackets,…
0
votes
1 answer

No module named 'gcp_sql_operator' in cloud composer

I am not able to import statement as- from airflow.contrib.operators.gcp_sql_operator import CloudSqlQueryOperator I want to import this in my DAG file which will be run in cloud composer airflow whose version is 1.10.0 and not 1.9.0.Here just to…
Aniruddha Dwivedi
  • 91
  • 1
  • 2
  • 13
0
votes
2 answers

Is there any way to edit an airflow operator after creation?

I have a python script that dynamically create task (airflow operator) and DAG basing on a JSON file that maps every option desired. The script also dedicated function to create any operator needed. Sometimes i want to activate some conditional…
aldroid
  • 25
  • 1
  • 8
0
votes
1 answer

Google Cloud Composer - Create Environment - with a few compute engine instances - That is expensive

I am new to Google Cloud Composer and following the QuickStart instruction, Create the Environment, Load Dag, Check Airflow, and Delete the Environment. But in (real life) production use case, after we finish load dag files and run them in the…
searain
  • 3,143
  • 6
  • 28
  • 60
0
votes
2 answers

Authorisation error when running airflow via cloud composer

I get an error when trying to run DAG from cloud composer using the GoogleCloudStorageToBigQueryOperator. Final error was: {'reason': 'invalid', 'location': 'gs://xxxxxx/xxxx.csv', and when I follow the URL link to the error ... { "error": { …
mez63
  • 166
  • 4
  • 20
0
votes
1 answer

Error accessing bucket using cloud composer

I'm getting error when trying to access a bucket using cloud composer... [2019-03-18 11:50:00,651] {models.py:1594} ERROR - 404 GET https://www.googleapis.com/storage/v1/b/gs://xxxx-cloud-composer?projection=noAcl: Not Found def…
mez63
  • 166
  • 4
  • 20
0
votes
1 answer

Economic possibility to execute many workflow tasks

so I have these Airflow DAGs which consists of several tasks. Basically each task executes some independent analysis steps against a given code snippet and finally it is decided if this snippet can be used from a regulatory point of view. Each…
pinas
  • 2,708
  • 4
  • 21
  • 33
0
votes
2 answers

Email on failure is not sent sendgrid/cloud composer

I've been using sendgrid to alert people on task failure in cloud composer. But email not been arriving to my email list even after failure also. Please find below default argument list default_dag_args = { "start_date": "2019-02-25", #…
0
votes
1 answer

Cannot run Airflow's KubernetesPodOperator using Default Cloud Composer Environment

Overview I just deployed a Cloud Composer environment using all defaults. I then followed this guide to try a simple KubernetesPodOperator out. When I copy the script into my DAGs directory and let it run, I get an error involving No SSH tunnels…
Tom
  • 554
  • 5
  • 22