How can I read multiple csv files from a gcs location, append them ( i.e. stack them ) write them back to another gcs location using DATA FUSION?

Question

Why Data Fusion, well coz I need to run several more steps ( run Data Proc clusters ) , insert to DBs and do it in a schedule. Also the data could explode ( 10s of TB ) or shrink ( 10s of GBs).

Answer 1

Stacking several TB files isn't a good idea. The Storage size limit per object is 5TB.

I don't know your need of stacking file.

Maybe Bigquery can be a solution for loading easily your CSV files and then to query subset of file for further processing. But querying 10s of TB is expensive! (5$ per TB)

For more help, add more detail on what you want to achieve.