Java Version Conflct with Protobuf

Question

I am creating a GRPC endpoint that will accept data and pass the data on to MapR Streams (port of Kafka). I have created the GPRC endpoint and that is working fine when dumping the data to console. When I add the code to push the messages to MapR/Kafka it compiles fine but when I run it I get an error relating to the protobuf library version. MapR is compiled against 2.5.0 where I am using 3.5.1 in my project. So is there a way for me to use the latest myself while getting the MapR libraries to use the older version? Note that in the past I have rolled the protobuf version back to 2.5.x and edited by ptoto files to be version 2 files. This is possible but painful as people send me v3 files usually.

My dependencies in maven are:

    <dependency>
        <groupId>com.google.protobuf</groupId>
        <artifactId>protobuf-java</artifactId>
        <version>3.5.1</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.grpc/grpc-stub -->
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-stub</artifactId>
        <version>1.30.2</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.grpc/grpc-protobuf -->
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-protobuf</artifactId>
        <version>1.31.0</version>
    </dependency>

    <!-- https://mvnrepository.com/artifact/io.grpc/grpc-netty -->
    <dependency>
        <groupId>io.grpc</groupId>
        <artifactId>grpc-netty</artifactId>
        <version>1.31.0</version>
    </dependency>

    <dependency>
        <groupId>com.mapr.streams</groupId>
        <artifactId>mapr-streams</artifactId>
        <version>6.1.0-mapr</version>
    </dependency>

Error is:

[d336599@mapredge01 ~]$ java -jar TLiveEndPoint-jar-with-dependencies.jar 
log4j:WARN No appenders could be found for logger (io.netty.util.internal.logging.InternalLoggerFactory).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
Aug 07, 2020 3:26:39 PM HelloWorldClient greet
INFO: Will try to greet world ...
Exception in thread "main" java.lang.RuntimeException: Error occurred while instantiating com.mapr.streams.impl.producer.MarlinProducerV10.
==> java.lang.VerifyError: class com.mapr.fs.proto.Marlinserver$MarlinInternalDefaults overrides final method getParserForType.()Lcom/google/protobuf/Parser;.
        at org.apache.kafka.clients.mapr.GenericHFactory.getImplementorInstance(GenericHFactory.java:42)
        at org.apache.kafka.clients.producer.KafkaProducer.initializeProducer(KafkaProducer.java:419)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:914)
        at org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:801)
        at MapRProducer.sendMessage(MapRProducer.java:16)
        at HelloWorldClient.greet(HelloWorldClient.java:51)
        at HelloWorldClient.main(HelloWorldClient.java:96)
Caused by: java.lang.reflect.InvocationTargetException
        at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
        at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
        at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
        at org.apache.kafka.clients.mapr.GenericHFactory.getImplementorInstance(GenericHFactory.java:39)
        ... 6 more
Caused by: java.lang.VerifyError: class com.mapr.fs.proto.Marlinserver$MarlinInternalDefaults overrides final method getParserForType.()Lcom/google/protobuf/Parser;
        at java.lang.ClassLoader.defineClass1(Native Method)
        at java.lang.ClassLoader.defineClass(ClassLoader.java:763)
        at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
        at java.net.URLClassLoader.defineClass(URLClassLoader.java:467)
        at java.net.URLClassLoader.access$100(URLClassLoader.java:73)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:368)
        at java.net.URLClassLoader$1.run(URLClassLoader.java:362)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.net.URLClassLoader.findClass(URLClassLoader.java:361)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at com.mapr.streams.impl.producer.MarlinProducerImpl.<init>(MarlinProducerImpl.java:73)
        at com.mapr.streams.impl.producer.MarlinProducerImplV10.<init>(MarlinProducerImplV10.java:27)
        at com.mapr.streams.impl.producer.MarlinProducerV10.<init>(MarlinProducerV10.java:10)
        ... 11 more

score 3 · Answer 1 · answered Nov 27 '20 at 18:06

We now shade the MapR libraries, with their old protobuf dependency, along with some of our convenience methods into our own 'table-access' JAR.

You have to include all of the dependencies that are in need of that protobuf dependency, because this is going to relocate the packages of the older protobuf classes, and rewrite the dependencies to refer to those relocated names.

Using Maven, and the maven-shade-plugin, that ends up looking like this:

  <build>
    <pluginManagement>
      <plugins>
        <plugin>
          <groupId>org.apache.maven.plugins</groupId>
          <artifactId>maven-shade-plugin</artifactId>
          <version>${maven-shade-plugin.version}</version>
          <executions>
            <execution>
              <id>shade-mapr-protobuf</id>
              <phase>package</phase>
              <goals>
                <goal>shade</goal>
              </goals>
              <configuration>
                <relocations>
                  <relocation>
                    <pattern>com.google.protobuf</pattern>
                    <shadedPattern>com.example.mapr.com.google.protobuf</shadedPattern>
                  </relocation>
                </relocations>
                <artifactSet>
                  <includes>
                    <include>com.mapr.fs:mapr-hbase</include>
                    <include>com.mapr.db:*</include>
                    <include>com.mapr.hadoop:maprfs</include>
                    <include>org.apache.hbase:*</include>
                    <include>org.apache.hadoop:*</include>
                    <include>com.google.protobuf:*</include>
                  </includes>
                </artifactSet>
                <filters>
                  <filter>
                    <artifact>*:*</artifact>
                    <excludes>
                      <exclude>org/apache/hadoop/yarn/util/package-info.class</exclude>
                      <exclude>org/apache/hadoop/yarn/factories/package-info.class</exclude>
                      <exclude>org/apache/hadoop/yarn/factory/providers/package-info.class</exclude>
                    </excludes>
                  </filter>
                </filters>
                <promoteTransitiveDependencies>true</promoteTransitiveDependencies>
                <keepDependenciesWithProvidedScope>true</keepDependenciesWithProvidedScope>
                <useBaseVersion>true</useBaseVersion>
              </configuration>
            </execution>
          </executions>
        </plugin>
      </plugins>
    </pluginManagement>
...
  </build>

Java Version Conflct with Protobuf

1 Answers1