-1

I'm trying to use cloud composer at few days But, I has a mission that spark submit to our existing Hadoop cluster with yarn mode

Is it possible using by cloud composer?

TJune
  • 1

1 Answers1

0

Anything that can be done by Apache Airflow 1.9.0 can be done by Cloud Composer currently. In this case, maybe take a look at the SparkSubmitOperator ?

cjmoberg
  • 106
  • 5
  • Hi Mittens thank for reply. I knew SparkSubmitOperator. but I think that spark submit on yarn operation available airflow worker inside yarn cluster server. but cloud composer was not accessing our yarn cluster using by SparkSubmitOperator – TJune Aug 09 '18 at 07:18