I'm trying to get https://scalapb.github.io/">ScalaPB to work with Spark. In order to do this I need to shade my protocol buffers within SBT using code like below
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.protobuf.**" -> "shadeproto.@1").inAll
)
This doesn't work within spark-shell. In fact, it doesn't even work within the Scala REPL. Reproduced below are my errors within the REPL.
Welcome to Scala 2.11.8 (Java HotSpot(TM) 64-Bit Server VM, Java 1.8.0_102).
Type in expressions for evaluation. Or try :help.
scala> val x = zefr.commons.proto.Video.VideoProto.descriptor
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'GeneratedMessageCompanion.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GeneratedMessageCompanion.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'GeneratedMessageCompanion.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GeneratedMessageCompanion.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term Descriptors in value com.protobuf,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.protobuf.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'VideoProto.class'.
Could not access type Descriptor in value com.Descriptors,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'VideoProto.class' was compiled against an incompatible version of com.Descriptors.
Here is a fully copy of my build.sbt file
scalaVersion := "2.11.8"
organization := "zefr.commons"
name := "commons-proto"
val nexus = "http://nexus.zefr.com/repository/maven"
resolvers ++= Seq(
Resolver.sonatypeRepo("releases"),
Resolver.sonatypeRepo("snapshots"),
"Neuxs Snapshots" at s"$nexus-snapshots",
"Neuxs Releases" at s"$nexus-releases"
)
conflictManager := ConflictManager.strict
PB.targets in Compile := Seq(
scalapb.gen() -> (sourceManaged in Compile).value
)
assemblyShadeRules in assembly := Seq(
ShadeRule.rename("com.google.protobuf.**" -> "shadeproto.@1").inAll
)
And project/protos.sbt
addSbtPlugin("com.thesamet" % "sbt-protoc" % "0.99.3")
libraryDependencies += "com.trueaccord.scalapb" %% "compilerplugin" % "0.5.45-p3"
addSbtPlugin("com.eed3si9n" % "sbt-assembly" % "0.14.3")
Can anyone please help me understand what I'm missing here?
Thanks
EDIT
In order to see if this was a problem that I was introducing I downloaded this project and built it as is, only changing the Scala version to 2.11.8 and I get the same error.
scala> val x = com.example.protos.demo.Person.descriptor
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'GeneratedMessageCompanion.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GeneratedMessageCompanion.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'GeneratedMessageCompanion.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'GeneratedMessageCompanion.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term Descriptors in value com.protobuf,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.protobuf.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term protobuf in value com.google,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.google.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access term google in package com,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.
error: missing or invalid dependency detected while loading class file 'Person.class'.
Could not access type Descriptor in value com.Descriptors,
because it (or its dependencies) are missing. Check your build definition for
missing or conflicting dependencies. (Re-run with `-Ylog-classpath` to see the problematic classpath.)
A full rebuild may help if 'Person.class' was compiled against an incompatible version of com.Descriptors.
