1

I am trying to create a pipeline in StreamSets Data Collector to read data from a Google Cloud Platform bucket and load the data into the same bucket with a different file name.

The data file in the bucket is in JSON form.

I used the Google Cloud Storage origin in StreamSets Data Collector and gave below properties:

  • Common Prefix = gs://<my-bucket-name>/<json-file-name>
  • Prefix Pattern = https://storage.cloud.google.com/<my-bucket-name>/<json-file-name>

Could someone correct or provide any alternative options?

metadaddy
  • 4,234
  • 1
  • 22
  • 46
Moushmi
  • 51
  • 9

1 Answers1

1

This is documented in Common Prefix, Prefix Pattern, and Wildcards.

  • Common prefix is a path common to all the files you want to read
  • Prefix pattern contains wildcards specifying the files you want to read

Neither of these should contain the bucket name (since that is configured separately) or the protocol. In your case, it looks like you can use something like:

  • Common prefix: /
  • Prefix pattern: *.json (or some other wildcard that matched your files)
metadaddy
  • 4,234
  • 1
  • 22
  • 46