0

I am newer in BigData, i have tried to call spark jobs with apache Livy . With submit command line works fine . with livy i have exception

  • the command line :

    curl -X POST --data '{"file": "/user/romain/spark-examples.jar", "className": "org.apache.spark.examples.SparkPi"}' -H 'Content-Type: application/json' http://localhost:8998/batches

  • Livy logs :

2019-06-01 00:43:19,160 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Exception in thread "main" java.io.FileNotFoundException: File hdfs://localhost:9000/home/spark-2.4.3-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.3.jar does not exist. at org.apache.hadoop.hdfs.DistributedFileSystem.listStatusInternal(DistributedFileSystem.java:795) .......org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:924) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:933) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)

OneCricketeer
  • 179,855
  • 19
  • 132
  • 245
EL missaoui habib
  • 1,075
  • 1
  • 14
  • 24
  • Does the jar that is mentioned in the error exist (and is it accessible at the specified location). – Dennis Jaheruddin Jun 03 '19 at 14:53
  • yes the jar exist and submit execute : ./bin/spark-submit --class org.apache.spark.examples.SparkPi --conf spark.executor.cores=2 examples/jars/spark-examples_2.11-2.4.3.jar 100 2019-06-03 22:55:19,494 WARN util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable Pi is roughly 3.1418867141886713 – EL missaoui habib Jun 03 '19 at 22:56
  • 2
    That command is using your local filesystem, not HDFS. Try `hadoop fs -ls hdfs://localhost:9000/home/spark-2.4.3-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.4.3.jar` to see the same exception – OneCricketeer Jun 04 '19 at 03:23
  • @cricket_007 the result is No such file or directory – EL missaoui habib Jun 04 '19 at 23:00
  • 1
    Right... So you need to do `hadoop fs -copyFromLocal` to copy from your local path into that HDFS path (and probably also `hadoop fs -mkdir /home` first) – OneCricketeer Jun 05 '19 at 01:56

2 Answers2

0

related to @cricket_007 comment resolved by execute hadoop fs -copyFromLocal command line

EL missaoui habib
  • 1,075
  • 1
  • 14
  • 24
0

If it is in local machine then try using 'file':'local:<path/to/file>

CHETAN RAJPUT
  • 131
  • 2
  • 14