0

I have a spark jar that I launch with spark-submit and it works fine (reading files, generate RDD, storing in hdfs). However, when I tried to launch the same jar within an Oozie job (oozie:spark-action) the spark job fails.

When I looked the logs, the first error to shows up is :

Error MetricsSystem: Sink class org.apache.spark.metrics.sink.MetricsServlet cannot be instantiated.

Furthermore, when I started playing with the spark script, I found out that the problem has to do with saveAsText funtion. When I lunch the same spark job without writing to HDFS the whole workflow works fine.

Any suggestions ?

OUMOUSS_ELMEHDI
  • 499
  • 5
  • 16

1 Answers1

0

The problem was in the side of the cluste where i am executing oozie jobs.

I needed to explicitely add arguments in the job workflow, simply because they weren't taken into consideration:

<spark-opts>--queue HQ_IBNF --conf "spark.executor.extraJavaOptions=-Djava.library.path=/opt/application/Hadoop/current/lib/native"</spark-opts>

OUMOUSS_ELMEHDI
  • 499
  • 5
  • 16