2

According to this page, runtime parameters are not supported for BigtableIO (only for BigQuery, PubSub and Text). Is there a possible workaround or example to do so without reimplementing the class?

Actually I was using CloudBigtableIO from bigtable-hbase-beam dependency. Is there an expectation on being supported on both libraries at some point?

Roberto Tena
  • 61
  • 3
  • If your goal is to write mutations to different tables at runtime, you may have to define a pipeline that is partitioned to write to the different tables. Is there any particular reason you need support for runtime parameters for Bigtable? – Andrew Nguonly Jan 30 '18 at 05:30
  • What's your goal? What do you want to do with runtime parameters? (I'm not sure what "runtime parameter" actually means in this case). I'm the maintainer of the library, so I'm happy to help, as long as I have more info. – Solomon Duskis Jan 30 '18 at 12:09
  • My main goal is to have the Bigtable instanceId and tableId configurable when running a job from a template. To run a job from a template you might be able to pass different parameters to make the IO parameters configurable, but they need to accept ValueProvider instances. To build the pipeline I used CloudBigtableIO.writeToTable() with a CloudBigtableTableConfiguration instance. As that configuration instance does not accept ValueProvider instances, I cannot do so. (continuing in the next comment) – Roberto Tena Jan 31 '18 at 09:55
  • During the day I have seen the [BigtableIO class](https://github.com/apache/beam/blob/master/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableIO.java) now accepts ValueProvider instances, but only in the master branch and it is not released yet. I personally found the hbase-beam library easier to manage, hence my particular interest in knowing more about this one. – Roberto Tena Jan 31 '18 at 09:56

1 Answers1

2

We now have parameters support, as of pull request https://github.com/apache/beam/pull/4357. Here is the code you may find interesting.

The documentation page you referenced also has the updated description for Bigtable.

Bonan Liu
  • 86
  • 8