1

I want to write a file to a GCS bucket. The bucket path and file name are dynamically provided in two different pipeline options. How can I concatenate those in TextIO to write the file to the GCS bucket.

I tried doing this but no luck.

o.apply("Test:",TextIO.write()
                .to(options.getBucktName().toString()+options.getOutName().toString()));

where getOutName = test.txt and getBucktName = gs://bucket

Edit: Options are ValueProvider

pas
  • 109
  • 1
  • 13

2 Answers2

2

We have faced a similar situation in Dataflow Templates, and we have created DualInputNestedValueProvider to address this.

You can feed it with 2 ValueProviders (it could be RuntimeValueProvider and a StaticValueProvider, in your case), and a function to map them to a new ValueProvider.

Take a look here for an example.

Bruno Volpato
  • 1,382
  • 10
  • 18
1

By "dynamically provided" do you mean those options are runtime ValueProvider instances? If so, I don't think it's possible to express what you want, since there's currently no hook for combining value providers (per related question).

If these are not value providers, then the example you show should work fine (although missing a / between the bucket and path as written).

Can you share more about how the options are defined?

Jeff Klukas
  • 1,259
  • 11
  • 20