Following steps for Hadoop hdfs and spark:-
1) Environment variables -
- HADOOP_CONF_DIR - F:\spark\Hadoop2\hadoop-2.7.6\etc\hadoop
- HADOOP_HOME - F:\spark\Hadoop2\hadoop-2.7.6
- JAVA_HOME - F:\Java\jdk1.8.0_121
- SPARK_HOME - F:\spark\spark-2.3.0-bin-hadoop2.7
Path:
- F:\spark\Hadoop2\hadoop-2.7.6\bin
- F:\spark\spark-2.3.0-bin-hadoop2.7\bin
- F:\spark\Hadoop2\hadoop-2.7.6\lib\native
2) core-site.xml
3) hdfs-site.xml
4) mapred-site.xml
Command to start master -
- spark-class org.apache.spark.deploy.master.Master
Command to start one worker-
- spark-class org.apache.spark.deploy.worker.Worker spark://192.168.0.12:7077
To submit job
- spark-submit --class org.apache.spark.examples.SparkPi --master spark://192.168.0.12:7077 F:/spark/spark-2.3.0-bin-hadoop2.7/examples/jars/spark-examples_2.11-2.3.0.jar
Error log -
Exception in thread "main" java.io.IOException: No FileSystem for scheme: F at org.apache.hadoop.fs.FileSystem.getFileSystemClass(FileSystem.java:2660) at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2667) at org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:94) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2703) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2685) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:373) at org.apache.spark.util.Utils$.getHadoopFileSystem(Utils.scala:1893) at org.apache.spark.util.Utils$.doFetchFile(Utils.scala:690) at org.apache.spark.deploy.DependencyUtils$.downloadFile(DependencyUtils.scala:131) at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:401) at org.apache.spark.deploy.SparkSubmit$$anonfun$prepareSubmitEnvironment$7.apply(SparkSubmit.scala:401) at scala.Option.map(Option.scala:146) at org.apache.spark.deploy.SparkSubmit$.prepareSubmitEnvironment(SparkSubmit.scala:400) at org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:170) at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:136) at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
I am not able to resolve the issue even after trying several blogs.
