So, I am running into the same issue that many are experiencing - see the error below.
WARN scheduler.TaskSetManager: Lost task 0.0 in stage 3.0 (): java.io.IOException: java.lang.reflect.InvocationTargetException
Caused by: java.lang.reflect.InvocationTargetException
Caused by: java.lang.NoClassDefFoundError: org/apache/htrace/Trace
Caused by: java.lang.ClassNotFoundException: org.apache.htrace.Trace
The instructions for fixing this seem to be missing, incomplete, or overly complex. I've resolved other reference issues simply by adding them to the packages list in spark-submit. I've tried doing the same with this htrace dependency too, but with no luck.
This is my current spark-submit command:
spark-submit --class com.biz.HbasePassthrough \
--packages \
org.apache.spark:spark-streaming-kafka_2.10:1.3.0,org.apache.hbase:hbase-common:1.0.0,org.apache.hbase:hbase-client:1.0.0,org.apache.hbase:hbase-server:1.0.0,org.json4s:json4s-jackson_2.10:3.2.11,org.apache.htrace:htrace:3.1.0-incubating,org.apache.htrace:htrace-core:3.1.0-incubating,org.apache.htrace:htrace-hbase:3.1.0-incubating,org.apache.hbase:hbase-annotations:1.0.0 \
./my-spark_2.10-1.0.8.jar \
>spark_log 2>&1
Output of grep -n1 "htrace" spark_log
8-org.json4s#json4s-jackson_2.10 added as a dependency
9:org.apache.htrace#htrace added as a dependency
10:org.apache.htrace#htrace-core added as a dependency
11:org.apache.htrace#htrace-hbase added as a dependency
12-org.apache.hbase#hbase-annotations added as a dependency
--
36- found org.mortbay.jetty#jetty-util;6.1.26 in central
37: found org.apache.htrace#htrace-core;3.1.0-incubating in central
38- found org.apache.hbase#hbase-client;1.0.0 in central
--
75- found com.fasterxml.jackson.core#jackson-core;2.3.1 in central
76: found org.apache.htrace#htrace;3.1.0-incubating in central
77: found org.apache.htrace#htrace-hbase;3.1.0-incubating in central
78-:: resolution report :: resolve 6423ms :: artifacts dl 45ms
--
111- org.apache.hbase#hbase-server;1.0.0 from central in [default]
112: org.apache.htrace#htrace;3.1.0-incubating from central in [default]
113: org.apache.htrace#htrace-core;3.1.0-incubating from central in [default]
114: org.apache.htrace#htrace-hbase;3.1.0-incubating from central in [default]
115- org.apache.kafka#kafka_2.10;0.8.1.1 from central in [default]
--
182-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/json4s-jackson_2.10.jar at http://192.168.240.209:37644/jars/json4s-jackson_2.10.jar with timestamp 1448926760838
183:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-core.jar at http://192.168.240.209:37644/jars/htrace-core.jar with timestamp 1448926760845
184:15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/htrace-hbase.jar at http://192.168.240.209:37644/jars/htrace-hbase.jar with timestamp 1448926760846
185-15/11/30 23:39:20 INFO spark.SparkContext: Added JAR file:/home/ubuntu/.ivy2/jars/hbase-annotations.jar at http://192.168.240.209:37644/jars/hbase-annotations.jar with timestamp 1448926760847
All of that output at the beginning of the job seems to suggest that it found the package and will use it successfully, but it doesn't work.
I did finally find something that I think got me past the issue - here. I adapted this by adding the following two lines to my spark-submit command above
--driver-class-path "/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \
--conf "spark.executor.extraClassPath=/opt/cloudera/parcels/CDH/lib/hbase/lib/htrace-core-3.1.0-incubating.jar" \
Why didn't the packages reference work?
