0

i have done ETL from MySql to bigQuery with python, but because i haven't permission to connect google cloud storage/ cloud sql, i must dump data and partition that by last date, this way easy but didn't worth it because take a much time, i want to ETL using airflow from MySql/mongo to bigQuery without google cloud storage/ cloud sql it is possible ?

Michael Hampton
  • 244,070
  • 43
  • 506
  • 972
user546730
  • 1

1 Answers1

0

Apache Airflow (which is used under Cloud Composer in Google Cloud) is generally used to schedule a migration. The actual migration along the complete ETL procedure can be performed with Cloud Dataflow (which uses Apache Beam) and is the recommended tool for this use case. Here is a complete tutorial for performing a ETL procedure from a relational database (such as MySQL) into BigQuery using Cloud Dataflow.

PYB
  • 101
  • 1
  • thanks for your suggestions, how about if my source from mongodb and mysql ? can i keep doing this tutorial ? @PYB – user546730 Nov 26 '19 at 02:00
  • The tutorial is valid for any relational database, such as MySQL. For MongoDB, see all the options for loading data into BigQuery here (https://cloud.google.com/bigquery/docs/loading-data), which will guide you in building your process. – PYB Nov 26 '19 at 15:03