0

My platform: Windows 7, 64 bits My Python: python 2.7 My Java: Java 1.7.0_79

I am following this guideline. https://spark.apache.org/docs/1.6.2/building-spark.html

I have download the package of Spark1.6.2. And I run this command successfully.

build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package

However, When I run

./make-distribution.sh --name custom-spark --tgz -Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn

I get error messages:

Spark version is 1.6.3-SNAPSHOT
Making spark-1.6.3-SNAPSHOT-bin-custom-spark.tgz
Tachyon Disabled

Building with...
$ /d/Python/spark/build/mvn clean package -DskipTests -Psparkr -Phadoop-2.4 -Phive -Phive-thriftserver -Pyarn

[INFO] Scanning for projects...
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Build Order:
[INFO] 
[INFO] Spark Project Parent POM
[INFO] Spark Project Test Tags
[INFO] Spark Project Launcher
[INFO] Spark Project Networking
[INFO] Spark Project Shuffle Streaming Service
[INFO] Spark Project Unsafe
[INFO] Spark Project Core
[INFO] Spark Project Bagel
[INFO] Spark Project GraphX
[INFO] Spark Project Streaming
[INFO] Spark Project Catalyst
[INFO] Spark Project SQL
[INFO] Spark Project ML Library
[INFO] Spark Project Tools
[INFO] Spark Project Hive
[INFO] Spark Project Docker Integration Tests
[INFO] Spark Project REPL
[INFO] Spark Project YARN Shuffle Service
[INFO] Spark Project YARN
[INFO] Spark Project Hive Thrift Server
[INFO] Spark Project Assembly
[INFO] Spark Project External Twitter
[INFO] Spark Project External Flume Sink
[INFO] Spark Project External Flume
[INFO] Spark Project External Flume Assembly
[INFO] Spark Project External MQTT
[INFO] Spark Project External MQTT Assembly
[INFO] Spark Project External ZeroMQ
[INFO] Spark Project External Kafka
[INFO] Spark Project Examples
[INFO] Spark Project External Kafka Assembly
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Parent POM 1.6.3-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ spark-parent_2.10 ---
[INFO] Deleting D:\Python\spark\target
[INFO] Deleting D:\Python\spark\lib_managed (includes = [], excludes = [])
[INFO] 
[INFO] --- maven-enforcer-plugin:1.4.1:enforce (enforce-versions) @ spark-parent_2.10 ---
[INFO] 
[INFO] --- scala-maven-plugin:3.2.2:add-source (eclipse-add-source) @ spark-parent_2.10 ---
[INFO] Add Source directory: D:\Python\spark\src\main\scala
[INFO] Add Test Source directory: D:\Python\spark\src\test\scala
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default-cli) @ spark-parent_2.10 ---
[INFO] Dependencies classpath:
C:\Users\childishwitch\.m2\repository\org\spark-project\spark\unused\1.0.0\unused-1.0.0.jar
[INFO] 
[INFO] --- maven-remote-resources-plugin:1.5:process (default) @ spark-parent_2.10 ---
[INFO] 
[INFO] --- scala-maven-plugin:3.2.2:compile (scala-compile-first) @ spark-parent_2.10 ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-antrun-plugin:1.8:run (create-tmp-dir) @ spark-parent_2.10 ---
[INFO] Executing tasks

main:
    [mkdir] Created dir: D:\Python\spark\target\tmp
[INFO] Executed tasks
[INFO] 
[INFO] --- scala-maven-plugin:3.2.2:testCompile (scala-test-compile-first) @ spark-parent_2.10 ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-dependency-plugin:2.10:build-classpath (default) @ spark-parent_2.10 ---
[INFO] 
[INFO] --- scalatest-maven-plugin:1.0:test (test) @ spark-parent_2.10 ---
[INFO] Tests are skipped.
[INFO] 
[INFO] --- maven-jar-plugin:2.6:test-jar (prepare-test-jar) @ spark-parent_2.10 ---
[INFO] Building jar: D:\Python\spark\target\spark-parent_2.10-1.6.3-SNAPSHOT-tests.jar
[INFO] 
[INFO] --- maven-site-plugin:3.3:attach-descriptor (attach-descriptor) @ spark-parent_2.10 ---
[INFO] 
[INFO] --- maven-shade-plugin:2.4.1:shade (default) @ spark-parent_2.10 ---
[INFO] Including org.spark-project.spark:unused:jar:1.0.0 in the shaded jar.
[INFO] Replacing original artifact with shaded artifact.
[INFO] 
[INFO] --- maven-source-plugin:2.4:jar-no-fork (create-source-jar) @ spark-parent_2.10 ---
[INFO] 
[INFO] --- maven-source-plugin:2.4:test-jar-no-fork (create-source-jar) @ spark-parent_2.10 ---
[INFO]                                                                         
[INFO] ------------------------------------------------------------------------
[INFO] Building Spark Project Test Tags 1.6.3-SNAPSHOT
[INFO] ------------------------------------------------------------------------
[INFO] 
[INFO] --- maven-clean-plugin:2.6.1:clean (default-clean) @ spark-test-tags_2.10 ---
[INFO] Deleting D:\Python\spark\tags\target
[INFO] ------------------------------------------------------------------------
[INFO] Reactor Summary:
[INFO] 
[INFO] Spark Project Parent POM ........................... SUCCESS [  7.964 s]
[INFO] Spark Project Test Tags ............................ FAILURE [  1.438 s]
[INFO] Spark Project Launcher ............................. SKIPPED
[INFO] Spark Project Networking ........................... SKIPPED
[INFO] Spark Project Shuffle Streaming Service ............ SKIPPED
[INFO] Spark Project Unsafe ............................... SKIPPED
[INFO] Spark Project Core ................................. SKIPPED
[INFO] Spark Project Bagel ................................ SKIPPED
[INFO] Spark Project GraphX ............................... SKIPPED
[INFO] Spark Project Streaming ............................ SKIPPED
[INFO] Spark Project Catalyst ............................. SKIPPED
[INFO] Spark Project SQL .................................. SKIPPED
[INFO] Spark Project ML Library ........................... SKIPPED
[INFO] Spark Project Tools ................................ SKIPPED
[INFO] Spark Project Hive ................................. SKIPPED
[INFO] Spark Project Docker Integration Tests ............. SKIPPED
[INFO] Spark Project REPL ................................. SKIPPED
[INFO] Spark Project YARN Shuffle Service ................. SKIPPED
[INFO] Spark Project YARN ................................. SKIPPED
[INFO] Spark Project Hive Thrift Server ................... SKIPPED
[INFO] Spark Project Assembly ............................. SKIPPED
[INFO] Spark Project External Twitter ..................... SKIPPED
[INFO] Spark Project External Flume Sink .................. SKIPPED
[INFO] Spark Project External Flume ....................... SKIPPED
[INFO] Spark Project External Flume Assembly .............. SKIPPED
[INFO] Spark Project External MQTT ........................ SKIPPED
[INFO] Spark Project External MQTT Assembly ............... SKIPPED
[INFO] Spark Project External ZeroMQ ...................... SKIPPED
[INFO] Spark Project External Kafka ....................... SKIPPED
[INFO] Spark Project Examples ............................. SKIPPED
[INFO] Spark Project External Kafka Assembly .............. SKIPPED
[INFO] ------------------------------------------------------------------------
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 10.672 s
[INFO] Finished at: 2016-07-06T12:03:53+08:00
[INFO] Final Memory: 38M/225M
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal org.apache.maven.plugins:maven-clean-plugin:2.6.1:clean (default-clean) on project spark-test-tags_2.10: Failed to clean project: Failed to delete D:\Python\spark\tags\target\spark-test-tags_2.10-1.6.3-SNAPSHOT.jar -> [Help 1]
[ERROR] 
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.
[ERROR] 
[ERROR] For more information about the errors and possible solutions, please read the following articles:
[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException
[ERROR] 
[ERROR] After correcting the problems, you can resume the build with the command
[ERROR]   mvn <goals> -rf :spark-test-tags_2.10

And after I doing so, whatever I run build/mvn -Pyarn -Phadoop-2.4 -Dhadoop.version=2.4.0 -DskipTests clean package or build/mvn clean install It will get the same error.

If I just type ./bin/pyspark It will get nothing and just done. I just want to import pyspark.mllib to use, but I don't know how to do so.

Because the limit of words amount, I paste the error message into this file: enter link description here
Hope someone can help me.

childishwitch
  • 23
  • 5
  • I forgot to say that I was running command under branch-1.6. – childishwitch Jul 06 '16 at 04:22
  • The test step is skipped, then later on info on spark test tags states FAILURE and the object `D:\Python\spark\tags\target\spark-test-tags_2.10-1.6.3-SNAPSHOT.jar` is not found or otherwise not deletable? Could you use the hints at the end of the output for using `-e` switch or even the debug hints, or are these not applicable? – Dilettant Jul 06 '16 at 05:11
  • Where's the test step? I tried to delete'D:\Python\spark\tags\target\spark-test-tags_2.10-1.6.3-SNAPSHOT.jar' manually, and it said the file is executing, and it can't be deleted. – childishwitch Jul 06 '16 at 08:32
  • I just read the provided output, and there it is printed, but I do no have access to the system ;-) nor do I know the components or libraries ... – Dilettant Jul 06 '16 at 09:47
  • I cloned the git using 'git clone git://github.com/apache/spark.git', following this website [http://spark.apache.org/downloads.html](http://spark.apache.org/downloads.html) – childishwitch Jul 07 '16 at 04:03
  • I am trying to run pyspark in Ubuntu now. It install pyspark successfully. However, I am trapped in linking Spark to python. [http://stackoverflow.com/questions/38323267/running-pyspark-mllib-on-ubuntu](http://stackoverflow.com/questions/38323267/running-pyspark-mllib-on-ubuntu) – childishwitch Jul 12 '16 at 08:23

0 Answers0