Data Fusion: GCS create creating folders not object

Question

I am trying to create an GCS object (File) with GCS create plugin of Data Fusion.

but it is creating a folder instead.

How I can have a file created instead of a folder ??

Answer 1

It seems that the description of the plugin leads to a misunderstanding. Cloud Storage doesn't work like a conventional filesystem, so you cannot "strictly" create empty files. The gsutil command doesn't have an equivalent to a touch command (on Linux) and all "basic" operation in this product is limited to the cp command (upload and download files).

Therefore, since there is no file when you specify the storage url, it's expected that a folder will be created instead of a file.

Based on this, I would like to suggest you two workarounds:

1. If you are using this plugin to create a file as a ‘flag’, you can continue using the plugin since the created folder also serves as a flag (to trigger a Cloud Function, for example)

2. If you need to create a file, you can create files with the GCS plugin located in ‘Sink’ plugins group to write records to one or more files in a directory on Google Cloud Storage. Files can be written in various formats such as csv, avro, parquet, and json.