Hadoop and Druid incompatibility issue with Jackson library

Question

I am running druid 0.9.0 on an Azure Cluster with HDP insight 2.4.1.1-3. The hadoop client is 2.7.1. After countless attempts to solve the issue with Jackson, specifically :

Error: class com.fasterxml.jackson.datatype.guava.deser.HostAndPortDeserializer overrides final method deserialize.

I 've tried all workarounds documented here

https://github.com/druid-io/druid/blob/master/docs/content/operations/other-hadoop.md

to no avail.

The recompilation was unsuccessful after shading the jackson dependency as suggested by another user facing the same issue.

The last case i tried was adding the

"mapreduce.job.user.classpath.first": "true"

to the jobProperties property of of my indexing task with the following results:

Diagnostics: Exception from container-launch.
Container id: container_e02_1461544451524_0047_05_000001
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
    at org.apache.hadoop.util.Shell.runCommand(Shell.java:576)
    at org.apache.hadoop.util.Shell.run(Shell.java:487)
    at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:753)
    at org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:371)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:303)
    at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
    at java.util.concurrent.FutureTask.run(FutureTask.java:262)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)
Shell output: main : command provided 1
main : run as user is nobody
main : requested yarn user is druid

We are trying to run an indexing job.

Any help will be appreciated. Has anyone had any success in the same setup?

score 0 · Answer 1 · answered May 06 '16 at 04:43

What is happening is that the version of jackson-datatype-guava has different minor version than jackson-databind; probably guava one is newer. Versions should match, although some combinations will work; one you have doesn't. What likely happens is that some earlier version of jackson-databind has specific method as final, but limitation has been relaxed in a later version.

Hadoop and Druid incompatibility issue with Jackson library

1 Answers1