0

I have a dataflow job that scans two bigtables and then joins the Result of the scan by the row key.

I am trying to set the coder to HbaseResultCoder but I keep getting an error that its not the correct type

PCollection<KV<String, KV<Result, Result>>> rowsJoined = JoinerUtils.leftJoin(oldBtScan, newBtScan)
    .setCoder(KvCoder.of(StringUtf8Coder.of(), KvCoder.of(HBaseResultCoder.getInstance(), HBaseResultCoder.getInstance())));

Error is reason: no instance(s) of type variable(s) K exist so that HBaseResultCoder conforms to Coder<K>

Sakib
  • 1,503
  • 4
  • 26
  • 39

1 Answers1

0

I didn't find any documentation about JoinerUtils.leftJoin which I presume might be the cause of the error. Why don't you take advantage of Dataflow escalation capabilities and use the Bigtable connector? With it you can read the Big Tables you want to combine in different PCollections, then use the letfOuterJoin of Apache Beam to combine them. Be aware that both PCollection must be in the form leftOuterJoin(PCollection> leftCollection, PCollection> rightCollection, V2 nullValue).

Hope you find this workaround helpful.

IDMT
  • 166
  • 5