0

I am Trying to transfer multiple (.csv) files under a directory from Azure storage container to Google storage (as .txt files)through data fusion. From Data fusion, I can successfully transfer single file and converting it to .txt file as part of GCS Sink.

But when I am trying to transfer all the .csv files under azure's container to GCS, it s merging all the .csv files data and generating single .txt file at GCS.

Can some one help on how to transfer each file separately and converting it to txt at Sink side?

Srini V
  • 65
  • 1
  • 1
  • 8

1 Answers1

1

What you're seeing is expected behavior when using GCS sink.

You need an Azure to GCS copy action plugin, or more generally an HCFS to GCS copy action plugin. Unfortunately such a plugin doesn't already exist. You could consider writing one using https://github.com/data-integrations/example-action as a starting point.

user3126412
  • 326
  • 1
  • 4
  • Thanks for your inputs. Could you let me know, can GCP dataflow help in achieving my requirement? Thanks – Srini V May 01 '21 at 08:35
  • Not sure about dataflow, but if you're looking for other ways to transfer data from Azure to GCS, take a look at https://stackoverflow.com/questions/41157327/transfer-file-from-azure-blob-storage-to-google-cloud-storage-programmatically. – user3126412 May 14 '21 at 00:01