1

I am trying to run a Spring Boot application on a Spark master using the following command:

spark-submit --master spark://IP-Address:Port --conf spark.executor.userClassPathFirst=true --conf spark.driver.userClassPathFirst=true --executor-memory 20G --total-executor-cores 100 JAR_PATH

The pom.xml is:

<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0"
    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>2.0.4.RELEASE</version>
        <relativePath /> <!-- lookup parent from repository -->
    </parent>
    <groupId>com.example.spark</groupId>
    <artifactId>spark-rdd-join-example</artifactId>
    <version>0.0.1-SNAPSHOT</version>
    <name>Spark RDD Join</name>
    <description>Demo project for RDD Spark Join</description>

    <properties>
        <java.version>1.8</java.version>
    </properties>

    <dependencies>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-jetty</artifactId>
        </dependency>

        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>


        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter</artifactId>
            <exclusions>
                <exclusion>
                    <groupId>org.springframework.boot</groupId>
                    <artifactId>spring-boot-starter-logging</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <dependency>
            <groupId>com.google.guava</groupId>
            <artifactId>guava</artifactId>
            <version>19.0</version>
        </dependency>


        <!-- Tests -->
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-test</artifactId>
            <scope>test</scope>
        </dependency>

        <dependency>
            <groupId>org.projectlombok</groupId>
            <artifactId>lombok</artifactId>
            <scope>provided</scope>
        </dependency>

        <dependency>
            <groupId>javax</groupId>
            <artifactId>javaee-api</artifactId>
            <version>8.0</version>
            <scope>provided</scope>
        </dependency>


        <dependency>
            <groupId>javax.validation</groupId>
            <artifactId>validation-api</artifactId>
        </dependency>


        <!-- Spark core -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-core_2.11</artifactId>
            <version>2.4.0</version>
            <exclusions>
                <exclusion>
                    <groupId>org.slf4j</groupId>
                    <artifactId>slf4j-log4j12</artifactId>
                </exclusion>
                <exclusion>
                    <groupId>log4j</groupId>
                    <artifactId>log4j</artifactId>
                </exclusion>
            </exclusions>
        </dependency>

        <!-- Spark Sql -->
        <dependency>
            <groupId>org.apache.spark</groupId>
            <artifactId>spark-sql_2.11</artifactId>
            <version>2.4.0</version>
        </dependency>

        <!--Spark Cassandra Connector -->
        <dependency>
            <groupId>com.datastax.spark</groupId>
            <artifactId>spark-cassandra-connector_2.11</artifactId>
            <version>2.4.0</version>
        </dependency>


        <!-- Needed for the Cassandra connection with Spark, otherwise connection 
            error -->
        <dependency>
            <groupId>io.dropwizard.metrics</groupId>
            <artifactId>metrics-core</artifactId>
            <version>3.2.2</version>
        </dependency>

    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.springframework.boot</groupId>
                <artifactId>spring-boot-maven-plugin</artifactId>
                <configuration>
                    <mainClass>com.example.spark.join.SparkRddJoinApplication</mainClass>
                </configuration>
            </plugin>
        </plugins>
    </build>

</project>

I am receiving the following error:

org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'defaultValidator' defined in class path resource [org/springframework/boot/autoconfigure/validation/ValidationAutoConfiguration.class]: Invocation of init method failed; nested exception is java.lang.NoSuchMethodError: javax.validation.BootstrapConfiguration.getClockProviderClassName()Ljava/lang/String;
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.jav
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:
 at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:49
 at org.springframework.beans.factory.support.AbstractBeanFactory.lambda$doGetBean$0(AbstractBeanFactory.java:317)
 at org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:222)
 at org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:315)
 at org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:199)
 at org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:759)
 at org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:869
 at org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:550)
 at org.springframework.boot.SpringApplication.refresh(SpringApplication.java:762)t.refresh(ServletWebServerApplicationContext.java:14
 at org.springframework.boot.SpringApplication.run(SpringApplication.java:330)n.java:398)
 at org.springframework.boot.SpringApplication.run(SpringApplication.java:1246)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)dJoinApplication.java:22)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at org.springframework.boot.loader.MainMethodRunner.run(MainMethodRunner.java:48)
 at org.springframework.boot.loader.Launcher.launch(Launcher.java:50)
 at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)r.java:51)
 at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
 at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:167)t$$runMain(SparkSubmit.scala:849)
 at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:86)
 at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933).scala:924)
 at org.hibernate.validator.internal.xml.config.ValidationBootstrapParameters.<init>(ValidationBootstrapParameters.java:63)

Caused by at org.hibernate.validator.internal.engine.ConfigurationImpl.parseValidationXml(ConfigurationImpl.java:540)ng/String;
        at org.springframework.validation.beanvalidation.LocalValidatorFactoryBean.afterPropertiesSet(LocalValidatorFactoryBean.java:309)
        at org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1695)58)
        ... 36 more

I already searched details about this error, but I still haven't found a solution for it. I already tried adding and removing dependencies, I added spark configuration properties.

It seems that I am stuck at this error.

Did someone deal with this problem and have a solution? Please help...

I saw some solutions related with Hibernate dependencies, but I am not using Hibernate in my project...

I am trying to perform a join over two Cassandra tables with Spark...

GritcoAndreea
  • 85
  • 11
  • Please post your Hibernate config. – mentallurg Sep 11 '19 at 19:21
  • I do not work with Hibernate in my project... – GritcoAndreea Sep 11 '19 at 19:24
  • See the stack trace: `org.hibernate.validator.internal.engine.ConfigurationImpl.parseValidationXml`. Obviously you are using Hibernate, directly or indirectly. If indirectly, may be it it used in Spark. Check Spark configs. – mentallurg Sep 11 '19 at 19:27
  • Oh, yes! I found it... It was a dependency express through spring-boot-starter-web.. I excluded it, but now it gives me the following error: Description: The Bean Validation API is on the classpath but no implementation could be found Action: Add an implementation, such as Hibernate Validator, to the classpath .....It somehow fixed the previous error but created another one.. – GritcoAndreea Sep 11 '19 at 19:42
  • What have you removed? Try to understand what are you doing, why do you have particular library, why do you have this library in a particular version. – mentallurg Sep 11 '19 at 19:49
  • The solution to this issue can be found [in this other question](https://stackoverflow.com/questions/47947293/maven-using-wrong-version-of-javax-validation) – Edu Castrillon Aug 17 '20 at 09:58

1 Answers1

0

You have compiled your code with Java EE 8, but trying to run on in Java EE environment. The interface BootstrapConfiguration existed also in the Java EE 7, but the method getClockProviderClassName was added only in Java EE 8.

mentallurg
  • 4,967
  • 5
  • 28
  • 36