5

I am getting started with Spark.

I am getting an issue when starting spark.

I downloaded from spark official website, I am trying to quickstart from this https://spark.apache.org/docs/0.9.0/quick-start.html

I have downloaded, setup sparkhome., and when I tried running ./pyspark from bin directory I get this error : 

soundarya@soundarya:~/Downloads/spark-1.5.0/bin$ sudo ./pyspark
[sudo] password for soundarya: 
ls: cannot access /home/soundarya/Downloads/spark-1.5.0/assembly/target/scala-2.10: No such file or directory
Failed to find Spark assembly in /home/soundarya/Downloads/spark-1.5.0/assembly/target/scala-2.10.
You need to build Spark before running this program.
soundarya@soundarya:~/Downloads/spark-1.5.0/bin$

Can anyone help me to figure out this?

zero323
  • 322,348
  • 103
  • 959
  • 935
Soundarya Thiagarajan
  • 574
  • 2
  • 13
  • 31

3 Answers3

2

If you want to use source distribution you have to build Spark first. You can find details in the Building Spark.

Otherwise just download a binary distribution (select pre-built) from downloads:

enter image description here

Also don't use outdated documentation. You can use either the latest:

http://spark.apache.org/docs/latest

or version specific

http://spark.apache.org/docs/1.5.0

zero323
  • 322,348
  • 103
  • 959
  • 935
  • Hi, thanks for your reply. I tried running it. 15/11/09 14:28:03 ERROR NettyTransport: failed to bind to /192.168.0.109:0, shutting down Netty transport 15/11/09 14:28:03 ERROR SparkContext: Error initializing SparkContext. java.net.BindException: Failed to bind to: /192.168.0.109:0: Service 'sparkDriver' failed after 16 retries! py4j.protocol.Py4JJavaError: An error occurred while calling None.org.apache.spark.api.java.JavaSparkContext. : java.net.BindException: Failed to bind to: /192.168.0.109:0: Service 'sparkDriver' failed after 16 retries! – Soundarya Thiagarajan Nov 09 '15 at 09:00
  • I downloaded spark-1.5.0-bin-hadoop2.4/bin – Soundarya Thiagarajan Nov 09 '15 at 09:00
  • Whats in your conf/spark-env.sh file? Did you try opening up a spark-shell first? (bin/spark-shell) – AkhlD Nov 09 '15 at 09:01
  • When I tried running spark-shell I got error : OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000000bff80000, 716177408, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 716177408 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/soundarya/Downloads/spark-1.5.0-bin-hadoop2.4/bin/hs_err_pid16684.log soundarya@soundarya:~/Downloads/spark-1.5.0-bin-hadoop2.4/bin$ – Soundarya Thiagarajan Nov 09 '15 at 09:03
  • 1
    that's because you are not having sufficient memory on your machine. What is executor memory/driver memory that you are allocating? and how much memory are you having on your machine? – AkhlD Nov 09 '15 at 09:33
0

Download the pre-build version of spark like zero323 suggested. Also, It is not a good practice to run spark in sudo mode.

AkhlD
  • 2,596
  • 2
  • 16
  • 15
  • Hi, I tried downloading as suggested : spark-1.5.0-bin-hadoop2.4 (prebuild version)..but still i m held up with errors.. OpenJDK 64-Bit Server VM warning: INFO: os::commit_memory(0x00000007d5500000, 716177408, 0) failed; error='Cannot allocate memory' (errno=12) # # There is insufficient memory for the Java Runtime Environment to continue. # Native memory allocation (malloc) failed to allocate 716177408 bytes for committing reserved memory. # An error report file with more information is saved as: # /home/soundarya/Downloads/spark-1.5.0/hs_err_pid18440.log – Soundarya Thiagarajan Nov 09 '15 at 09:46
  • How much memory do you have on your machine? Its a JVM Heap error and has nothing to do with spark. You can try setting the spark.driver.memory and spark.executor.memory in the conf/spark-defaults.conf file. Or simply do a export _JAVA_OPTIONS=-Xmx1g from your terminal. – AkhlD Nov 09 '15 at 10:00
  • I did export _JAVA_OPTIONS=-Xmx1g ; Picked up _JAVA_OPTIONS: -Xmx1g raise Exception("Java gateway process exited before sending the driver its port number") Exception: Java gateway process exited before sending the driver its port number >>> # There is insufficient memory for the Java Runtime Environment to continue. – Soundarya Thiagarajan Nov 09 '15 at 10:09
  • "There is insufficient memory for the Java Runtime Environment to continue. "- How much memory are you having on your machine? – AkhlD Nov 09 '15 at 10:22
  • Hi AkhID, I have posted it as seperate answer. – Soundarya Thiagarajan Nov 09 '15 at 10:44
0

Check space in your spark folder name like spark 2.4 then remove it and make spark-2.4 will fix issue

vaquar khan
  • 10,864
  • 5
  • 72
  • 96