0

I am new to google cloud composer. I have some code in google cloud compute engine - for eg: test.py

Currently I am using Jenkins as my scheduler - and I'm running the code like below

echo "cd /home/user/src/digital_platform &&/home/user/venvs/bdp/bin/python -m test.test.test" | ssh user@instance-dp

I want to run the same code from google cloud composer.

How I can do that..

Basically I need to ssh to an instance in google cloud and run the code in an automated way using google cloud composer.

Hacker
  • 133
  • 5
  • 16
  • Check this out :https://stackoverflow.com/questions/39457592/airflow-how-to-ssh-and-run-bashoperator-from-a-different-server – ECris Jun 19 '19 at 19:37

2 Answers2

3

It seems that SSHOperator might be something that might work for you. This operator is an Airflow feature, not Cloud Composer feature per se.

The other operator that you might want to take a look at before making your final decision is BaskOperator

rafalbiegacz
  • 96
  • 5
0

You need to create a DAG (workflows), Cloud Composer schedules only the DAGs that are in the DAGs folder in the environment's Cloud Storage bucket. Each Cloud Composer environment has a web server that runs the Airflow web interface that you can use to manage DAGs.

Bash Operator is useful to run command-line programs. I suggest you follow the Cloud Composer Quickstart which shows you how to create a Cloud Composer environment in the Google Cloud Console and run a simple Apache Airflow DAG.

Enrique Zetina
  • 825
  • 5
  • 16