0

I know we have similar questions already answered here. but for some reason none of the options are working for me.

Below is the error i'm getting:

User class threw exception: java.lang.NoSuchMethodError: com.fasterxml.jackson.databind.introspect.AnnotatedMember.getType()Lcom/fasterxml/jackson/databind/JavaType;
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector._fullSerializationType(JaxbAnnotationIntrospector.java:1550)
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector._findContentAdapter(JaxbAnnotationIntrospector.java:1502)
at com.fasterxml.jackson.module.jaxb.JaxbAnnotationIntrospector.findSerializationContentConverter(JaxbAnnotationIntrospector.java:914)
at com.fasterxml.jackson.databind.introspect.AnnotationIntrospectorPair.findSerializationContentConverter(AnnotationIntrospectorPair.java:388)
at com.fasterxml.jackson.databind.ser.std.StdSerializer.findConvertingContentSerializer(StdSerializer.java:266)
at com.fasterxml.jackson.databind.ser.std.AsArraySerializerBase.createContextual(AsArraySerializerBase.java:197)

Below is the dependencies i have added. I tried removing all the jackson dependencies and added the required version of them seperately.

dependencies {
compile (group: 'org.scala-lang', name: 'scala-library', version: '2.11.8') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'org.apache.spark', name: 'spark-core_2.11', version: '2.3.0') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}
compile (group: 'org.apache.spark', name: 'spark-sql_2.11', version: '2.3.0') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'com.amazonaws', name: 'aws-java-sdk', version: '1.11.633') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile group: 'org.apache.httpcomponents', name: 'httpclient', version: '4.5.2'
compile group: 'com.sun.mail', name: 'javax.mail', version: '1.6.2'

testCompile group: 'junit', name: 'junit', version: '4.12'

compile group: 'net.liftweb', name: 'lift-json_2.11', version: '2.6.3'
compile group: 'net.sf.json-lib', name: 'json-lib', version: '2.4', classifier: 'jdk15'
compile group: 'net.sf.oval', name: 'oval', version: '1.31'

compile (group: 'com.bedatadriven', name: 'jackson-datatype-jts', version: '2.4') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

compile (group: 'com.fasterxml.jackson.datatype', name: 'jackson-datatype-joda', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile group: 'com.fasterxml.jackson.core', name: 'jackson-core', version: '2.11.2'
compile (group: 'com.fasterxml.jackson.core', name: 'jackson-databind', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile group: 'com.fasterxml.jackson.core', name: 'jackson-annotations', version: '2.11.2'

compile (group: 'com.fasterxml.jackson.module', name: 'jackson-module-jaxb-annotations', version: '2.11.2') {
    exclude group: 'com.fasterxml.jackson.core'
}

compile (group: 'com.fasterxml.jackson.dataformat', name: 'jackson-dataformat-csv', version: '2.11.2') {
     exclude group: 'com.fasterxml.jackson.core'
 }

// Some custom Jars

compile (group: 'org.springframework', name: 'spring-core', version: '2.5.6') {
    exclude group: 'com.fasterxml.jackson.module'
    exclude group: 'com.fasterxml.jackson.core'
    exclude group: 'com.fasterxml.jackson.dataformat'
    exclude group: 'com.fasterxml.jackson.datatype'
}

}

this configuration works when i run the spark job in my local system but not in aws emr cluster.

Rajashree Gr
  • 519
  • 1
  • 9
  • 18
  • Note that you should mark Spark and Scala library dependencies as `provided` – Gaël J Jun 01 '21 at 19:31
  • If I mark the Spark & Scala library dependencies as provided, then EMR provided Spark will be used right. I think I should try the below options which you have provided. – Rajashree Gr Jun 01 '21 at 20:20

1 Answers1

1

Spark provides Jackson itself, and not the version you expect. The error is likely caused by having 2 conflicting Jackson versions in the classpath.

You have 2 options:

Spark 2.3.0 comes with Jackson 2.6.7.1 (can be checked here for instance: https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11/2.3.0).

Gaël J
  • 11,274
  • 4
  • 17
  • 32
  • I excluded the Jackson dependencies from Spark which can be seen in the dependencies section. – Rajashree Gr Jun 01 '21 at 19:27
  • 1
    Yes but Spark will provide them anyway. You can see Spark as a runtime environment with a set of predefined dependencies which you cannot remove. – Gaël J Jun 01 '21 at 19:30
  • The first option didn't work for me. I'm trying to options now. the second option which has mentioned and the relocation using shadow jar – Rajashree Gr Jun 02 '21 at 15:08
  • 1
    Do you need a recent Jackson? It's obviously not a good solution but if you don't really need a recent Jackson, I would just re-use the Jackson version provided with your Spark version. _The 1st option should work though, I've already used it but not in the exact same context as you_ – Gaël J Jun 02 '21 at 15:39
  • Thanks a lot, Gael. I modified the code to use the Jackson version provided by the spark and it worked. – Rajashree Gr Jun 04 '21 at 03:00
  • I was also trying to use the shadow jar with relocate option in Gradle. For some reason, it is not working me. I was able to fix the code by lowering the Jackson version but curious about what is wrong with the shadow jar – Rajashree Gr Jun 04 '21 at 03:02
  • shadowJar { zip64 = true exclude 'META-INF/*.RSA' exclude 'META-INF/*.SF' exclude 'META-INF/*.DSA' archiveVersion.set(version) relocate 'com.fasterxml.jackson.core:jackson-databind', 'shadow.com.company.fasterxml.jackson.core:jackson-databind' } – Rajashree Gr Jun 04 '21 at 03:02