3

I am planning to use Google Cloud Composer to schedule a workflow which is loading data from S3 to Redshift. As S3 and Redshift are all based on AWS, I want to know whether Google Cloud Composer allows me to do it.

From my understanding after reading the document, I am able to install AWS SDK from PYPI and install local Python dependencies, which means that I can build my own library for S3Connection and RedshiftConnection to make the workflow work. Anyone with experience on it can help confirm it? Thanks

yyyyyyyyoung
  • 305
  • 1
  • 10
  • Is the answer enough to answer your question? Just want to make sure.. – Pablo Jan 15 '19 at 18:32
  • I am considering a similar setup. To me, the biggest question is how to connect and authenticate between the two different clouds. Has anybody done this and if so how? – Stephen Jan 11 '20 at 19:08

1 Answers1

3

Cloud Composer is built to support hybrid and multi-cloud as mentioned here.

Cloud Composer can make use of existing Airflow operators which includes a S3toRedshift operator that uses the S3 hook and the postgres hook for Redshift

Cloud Composer can also use custom plugins. The way to install custom plugins in Cloud Composer is described in this link.

Kannappan Sirchabesan
  • 1,353
  • 11
  • 21