SBT Verify Error caused by multiple protobuf 2/3 dependencies in spite of shading

Question

I am struggling with Verify Errors with this below sample project using Scio/Bigtable/HBase. The dependency tree requires protobuf version (2.5, 2.6.1, 3.0, 3.1) and seems to default to 3.2. I used the shading component of sbt-assembly, not sure I am right with it.

My build.sbt:

name := "scalalab"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies ++= Seq(
  "com.spotify" %% "scio-core" % "0.3.2",
  "com.google.cloud.bigtable" % "bigtable-hbase-1.2" % "0.9.2",
  "org.apache.hbase" % "hbase-client" % "1.2.5",
  "org.apache.hbase" % "hbase-common" % "1.2.5",
  "org.apache.hadoop" % "hadoop-common" % "2.7.3"
)

assemblyShadeRules in assembly := Seq(
  ShadeRule.rename("com.google.**" -> "shadeio.@1").inAll
)

assemblyMergeStrategy in assembly := {
  case x => MergeStrategy.first
}

My Main.scala:

import com.google.cloud.bigtable.hbase.adapters.Adapters
import com.google.cloud.bigtable.hbase.adapters.read.DefaultReadHooks
import org.apache.hadoop.hbase.client.Scan

object Main {
  def main(args: Array[String]): Unit = {
    val scan = new Scan
    scan.setMaxVersions()
    scan.addFamily("family".getBytes)
    scan.setRowPrefixFilter("prefix".getBytes)

    val builder = Adapters.SCAN_ADAPTER.adapt(scan, new DefaultReadHooks)

    System.out.println(builder)
  }
}

outputs to:

scalalab git:(master) ? java -cp .:target/scala-2.11/scalalab-assembly-1.0.jar Main
Exception in thread "main" java.lang.VerifyError: Bad return type
Exception Details:
  Location:
    shadeio/cloud/bigtable/hbase/adapters/AppendAdapter.adapt(Lorg/apache/hadoop/hbase/client/Operation;)Lshadeio/bigtable/repackaged/com/google/protobuf/GeneratedMessageV3$Builder; @8: areturn
  Reason:
    Type 'shadeio/bigtable/v2/ReadModifyWriteRowRequest$Builder' (current frame, stack[0]) is not assignable to 'shadeio/bigtable/repackaged/com/google/protobuf/GeneratedMessageV3$Builder' (from method signature)
  Current Frame:
    bci: @8
    flags: { }
    locals: { 'shadeio/cloud/bigtable/hbase/adapters/AppendAdapter', 'org/apache/hadoop/hbase/client/Operation' }
    stack: { 'shadeio/bigtable/v2/ReadModifyWriteRowRequest$Builder' }
  Bytecode:
    0x0000000: 2a2b c000 28b6 00a0 b0

        at shadeio.cloud.bigtable.hbase.adapters.Adapters.<clinit>(Adapters.java:40)
        at Main$.main(Main.scala:12)
        at Main.main(Main.scala)

What am I doing wrong ?

Thanks for your help

Solomon Duskis · Answer 1 · 2017-06-27T21:13:43.333

1

The dependency issue is quit a pain for the Cloud Bigtable HBase client. We'll be fixing it in the next release. To fix your current problem, import "bigtable-hbase" instead of "bigtable-hbase-1.2"

Also, I would suggest using the latest version of the client is possible, which is 0.9.7.1.

edited Jun 27 '17 at 21:13

answered Jun 23 '17 at 20:35

Solomon Duskis

2,691
16
12

Thanks for your reply. With your dependency replacement, Adapters is no longer importable... – ogen Jun 26 '17 at 13:00
Ugh... Adapters is moved to a different package, and so do the protobuf objects. Sorry about that. You can use `bigtable-hbase` instead. – Solomon Duskis Jun 26 '17 at 13:06
Thanks for your reply, gets another error `java.lang.NoSuchMethodError: shadeio.bigtable.v2.RowRange$Builder.setStartKeyClosed(Lshadeio/bigtable/repackaged/com/google/protobuf/ByteString;)Lshadeio/bigtable/v2/RowRange$Builder`. This may be related to a conflit in 2+ versions of protobuf. – ogen Jun 27 '17 at 15:52
The "repacka‌ged" version of the protos is in `bigtable-hbase-1.2` and `bigtable-hbase-shaded-for-dataflow`. You probably should use `bigtable-hbase`. – Solomon Duskis Jun 27 '17 at 21:14
FYI, tomorrow (hopefully), we'll release a new version of the client that has an HBase Beam connector. – Solomon Duskis Jun 27 '17 at 21:15

score 0 · Answer 2 · answered Jun 23 '17 at 20:24

Scio has a bigtable artifact, which you can include via:

libraryDependencies ++= Seq(
  "com.spotify" %% "scio-core" % "0.3.2",
  "com.spotify" %% "scio-bigtable" % "0.3.2"
)

Usually, there is no need for the other dependencies to use Bigtable, in fact they might cause issues.

Make sure to then import bigtable package via:

import com.spotify.scio.bigtable._

Check this BigtableExample.

Side note you might want to give sbt-pack a try instead of sbt-assembly to fully leverage artifact caching. For an example of sbt setup check the template here.

Thanks for your reply. My post was not very clear, but in the above code, I used the Bigtable client outside of scio. Using the bigtable client through Scio does work with your dependency. — ogen, Jun 26 '17 at 13:03

SBT Verify Error caused by multiple protobuf 2/3 dependencies in spite of shading

2 Answers2