Questions tagged [google-cloud-composer]

Google Cloud Composer is a fully managed workflow orchestration service, built on Apache Airflow, that empowers you to author, schedule, and monitor pipelines that span across clouds and on-premises data centers.

Cloud Composer is a product of Google Cloud Platform (GCP). It is essentially "hosted/managed Apache Airflow."

The product allows you to create, schedule, and monitor jobs, each job being represented as a DAG (directed acyclic graph) of various operators. You can use Airflow's built-in operator definitions and/or or define your own in pure Python.

While technically you can do data processing directly within a task (instantiated operator), more often you will want a task to invoke some sort of processing in another system (which could be anything - a container, BigQuery, Spark, etc). Often you will then wait for that processing to complete using an Airflow sensor operator, possibly launch further dependent tasks, etc.

While Cloud Composer is managed, you can apply a variety of customizations, such as specifying which pip modules to install, hardware configurations, environment variables, etc. Cloud Composer allows overriding some but not all Airflow configuration settings.

Further technical details: Cloud Composer will create a Kubernetes cluster for each Airflow environment you create. This is where your tasks will be run, but you don't have to manage it. You will place your code within a specific bucket in Cloud Storage, and Cloud Composer will sync it from there.

1225 questions
0
votes
2 answers

on_failure_callback in dynamic dags

I'm creating dags dynamically from a list and would like to add an on_failure_callback to one of the tasks. I've tried the following code, but the callback doesn't seem to be getting executed. dag_ids = ['dag_a', 'dag_b', 'dag_c'] for dag_id in…
blessonm
  • 71
  • 5
  • 12
0
votes
1 answer

Cloud composer issue with datasets in Australia region

I was trying to use cloud composer to schedule and orchestrate Bigquery jobs. Bigquery tables are in australia-southeast1 region.The cloud composer environment was created in us-central1 region(As composer is not available in Australia region). When…
0
votes
1 answer

Passing runtime version using Airflow operator to CMLE

I'm using the MLEngine operator in contrib (could not find one in none contrib). The problem from the logs is runtime error which is not set properly and then defaults to 1.0. File "/root/.local/lib/python2.7/site-packages/trainer/model.py", line …
JY2k
  • 2,879
  • 1
  • 31
  • 60
0
votes
1 answer

MLengine 'module' object has no attribute 'estimator'

Running this example on ML engine using Cloud composer but am receiving the following error: AttributeError: 'module' object has no attribute 'estimator' Even though I am importing import tensorflow as tf and it exits on the following line: …
JY2k
  • 2,879
  • 1
  • 31
  • 60
0
votes
1 answer

Supplying arguments to MLEngine airflow operator

I have been able to successfully train using the gcloud ml-engine sumbit job CLI. I then copied the "args" value from the console training input of the success job. "args": [ "--output_dir=gs://composer-models/flowers/trained_cnn", …
JY2k
  • 2,879
  • 1
  • 31
  • 60
0
votes
2 answers

Airflow ML Engine Package_URI

What is the package URI used for? Is it mandatory? If so how do I create one? Currently I have my model package into the proper format of: model.py task.py _init_.py
JY2k
  • 2,879
  • 1
  • 31
  • 60
0
votes
1 answer

Import missing module

What is the convention for properly importing a module in Airflow? I would like to import this operator: https://github.com/apache/airflow/blob/master/airflow/contrib/operators/mlengine_operator.py using this line: from airflow.contrib.operators…
JY2k
  • 2,879
  • 1
  • 31
  • 60
0
votes
1 answer

How to run container with privileged flags using KubernetesPodOperator in GoogleCloudComposer?

I need to use a container with gcsfuse in google cloud composer, as far as i know --privileged flags when running the container is needed for gcsfuse, how to use --privileged flags when running the container using kubernetes pod operator in google…
0
votes
1 answer

ML-engine fails from composer -Unknown name "python-version"

Im trying to launch an ml-engine jobs submit training using a cloud composer, i'm using this guide for instructions recommendation-system-tensorflow-deploy. Im using a plugin which google created (see the implementation here) Im trying to make it…
Yehoshaphat Schellekens
  • 2,305
  • 2
  • 22
  • 49
0
votes
1 answer

Google Cloud Composer, airflow job cannot recognize installed PyPi packages

I am working on Airflow using Google Cloud Composer. Here is the dag file: from airflow import DAG from airflow.operators.bash_operator import BashOperator from datetime import datetime, timedelta dag = DAG( 'hello_world', …
0
votes
1 answer

How To Restart Self Managed Airflow Webserver in Google Composer

I'm using Google Composer however due to the fact that I'm trying to create a dynamically created DAG, I'm unable to use the managed web server [1]. So in the end I need to add a plugins because I need to access the Airflow Webserver using REST API,…
irvifa
  • 1,865
  • 2
  • 16
  • 20
0
votes
2 answers

Where is Airflow webserver running on Google Composer?

I have following pods: NAME READY STATUS RESTARTS AGE airflow-database-init-job-ggk95 0/1 Completed 0 3h airflow-redis-0 …
Bunyk
  • 7,635
  • 8
  • 47
  • 79
0
votes
1 answer

How to fetch configuration of Google Composer workflow from Cloud Datastore?

I want to get my DAG configuration from datastore, so in file that configures workflows I write following import: from google.cloud import datastore I got import error, so I went to PYPI packages tab for my envionmment and added…
0
votes
1 answer

Setting Airflow Environment Variables And Accessing Connection

We are trying to create an Airflow Connection by setting an environment in Cloud Composer: export…
0
votes
0 answers

How to log using custom metric in GCP Cloud Composer using python?

I want to log message with custom metric in google cloud composer's DAG. Assume that this metric is some number. This will allow me to write rule/policy with some threshold and further send email. Could you please provide python snippet to log with…
Omkara
  • 414
  • 4
  • 16