Libraries conflict after upgrade of google-cloud library in my java software running on dataproc

Question

I have a problem after upgrade google-cloud library from 0.8.0 to 0.32.0-alpha version on my java software running on google dataproc.

Here my maven dependencies:

<dependencies>
    <dependency>
        <groupId>com.google.cloud</groupId>
        <artifactId>google-cloud</artifactId>
        <version>0.32.0-alpha</version>  
    </dependency>

    <dependency>
        <groupId>org.apache.httpcomponents</groupId>
        <artifactId>httpclient</artifactId>
        <version>4.5.2</version>
    </dependency>

    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-api</artifactId>
        <version>1.7.19</version>
    </dependency>

    <dependency>
        <groupId>org.slf4j</groupId>
        <artifactId>slf4j-log4j12</artifactId>
        <version>1.7.19</version>
     </dependency>

     <dependency>
        <groupId>org.apache.tika</groupId>
        <artifactId>tika-core</artifactId>
        <version>1.12</version>
     </dependency>

     <dependency>
        <groupId>args4j</groupId>
        <artifactId>args4j</artifactId>
        <version>2.33</version>
     </dependency>

     <dependency>
        <groupId>junit</groupId>
        <artifactId>junit</artifactId>
        <version>4.12</version>
        <scope>test</scope>
     </dependency>

     <dependency>
        <groupId>com.googlecode.json-simple</groupId>
        <artifactId>json-simple</artifactId>
        <version>1.1</version>
     </dependency>

     <dependency>
        <groupId>org.mockito</groupId>
        <artifactId>mockito-all</artifactId>
        <version>2.0.2-beta</version>
        <scope>test</scope>
     </dependency>

     <dependency>
         <groupId>javax.mail</groupId>
         <artifactId>mail</artifactId>
         <version>1.4</version>
     </dependency>

     <dependency>
         <groupId>mysql</groupId>
         <artifactId>mysql-connector-java</artifactId>
         <version>5.1.39</version>
     </dependency>

     <dependency>
         <groupId>commons-lang</groupId>
         <artifactId>commons-lang</artifactId>
         <version>2.6</version>
     </dependency>

</dependencies>

This is the error I can see in the dataproc job output

Exception in thread "main" java.lang.NoSuchMethodError: com.google.common.util.concurrent.MoreExecutors.directExecutor()Ljava/util/concurrent/Executor;
    at com.google.api.gax.retrying.BasicRetryingFuture.<init>(BasicRetryingFuture.java:77)
    at com.google.api.gax.retrying.DirectRetryingExecutor.createFuture(DirectRetryingExecutor.java:73)
    at com.google.cloud.RetryHelper.run(RetryHelper.java:73)
    at com.google.cloud.RetryHelper.runWithRetries(RetryHelper.java:51)
    at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:375)
    at com.google.cloud.bigquery.BigQueryImpl.getTable(BigQueryImpl.java:366)
    at com.finscience.job.link.LinkAnalyzerProcess.process(LinkAnalyzerProcess.java:210)
    at com.finscience.job.link.LinkAnalyzerJob.main(LinkAnalyzerJob.java:140)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:755)
    at org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:180)
    at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:205)
    at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:119)
    at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

It seems that the problem is related to google.api.gax library.

After a search I found people that solved the problem excluding guava from maven dependencies. So I modified my google cloud dependency in this way:

<dependency>
    <groupId>com.google.cloud</groupId>
    <artifactId>google-cloud</artifactId>
    <version>0.32.0-alpha</version>
    <exclusions>
       <exclusion>
           <artifactId>com.google</artifactId>
            <groupId>guava</groupId>
       </exclusion>  
    </exclusions>  
 </dependency>

But unfortunately this do not solve my problem.

The github page of the google cloud library (https://github.com/googlecloudplatform/google-cloud-java) says that:

The easiest way to solve version conflicts is to use google-cloud's BOM

So I add the following dependency to my pom:

<dependency>
   <groupId>com.google.cloud</groupId>
   <artifactId>google-cloud-bom</artifactId>
   <version>0.41.0-alpha</version>
   <type>pom</type>
   <scope>import</scope>
</dependency>

but even in this case the problem has not been solved.

Could someone help me?

Answer 1

You are most likely running into conflicts with Spark and Hadoop jars on the Dataproc cluster. This answer goes into how to repackage jars to deal with Hadoop / Spark conflicts.