Using Google Cloud Dataflow for merging flat files and importing into Cloud SQL

Question

We have to read data from CSV files and map two files with respect to one column and push data to Cloud SQL using Google Cloud Dataflow.

We are able to read data from CSV files but stuck with the next steps. Please provide me information or links regarding the following:

Merging/joining to flat files based on one column or condition with multiple columns
Copying merged pcollection into Сloud SQL database

Can you please clarify what you mean by "map two files with respect to one column" - do you mean you want to join two CSV files based on the value of a column and then dump the result to Cloud SQL? A short example of input and expected output would be helpful. — jkff, Apr 24 '15 at 16:47

score 3 · Answer 1 · answered Apr 24 '15 at 17:08

Here's some pointers that may be helpful:

https://cloud.google.com/dataflow/model/joins describes the ways to join PCollection's in Dataflow
There is currently no built-in sink for writing to CloudSQL, however you can either simply process the results of your join using a ParDo which writes each individual record or in batches (flushing periodically or in finishBundle()) - or if your needs are more complex than that, consider writing a CloudSQL sink - see https://cloud.google.com/dataflow/model/sources-and-sinks

How do you configure the Cloud SQL instance for access from DataFlow? Do you need to somehow use the proxy? — sthomps, Apr 09 '16 at 04:18

1 Answers1