11

I am running a Spark program on Intellij and getting the below error : "object apache is not a member of package org".

I have used these import statement in the code : 

import org.apache.spark.SparkContext  
import org.apache.spark.SparkContext._  
import org.apache.spark.SparkConf

The above import statement is not running on sbt prompt too. The corresponding lib appears to be missing but I am not sure how to copy the same and at which path.

Learner
  • 113
  • 1
  • 1
  • 4

2 Answers2

9

Make sure you have entries like this in SBT:

scalaVersion := "2.11.8"
libraryDependencies ++= Seq(
  "org.apache.spark" %% "spark-core" % "2.1.0", 
  "org.apache.spark" %% "spark-sql" % "2.1.0" 
)

Then make sure IntelliJ knows about these libraries by either enabling "auto-import" or doing it manually by clicking the refresh-looking button on the SBT panel.

Vidya
  • 29,932
  • 7
  • 42
  • 70
  • I tried by adding the above librarydependencies too but still, it is not working : [error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.spark #spark-core_2.12;2.1.0: not found – Learner Apr 09 '17 at 17:14
  • 2
    Change your Scala version to `2.11.8` – Vidya Apr 09 '17 at 17:24
  • Thanks, it helped ! – Learner Apr 13 '17 at 05:05
  • Glad to hear it. Good luck with your project! – Vidya Apr 13 '17 at 12:34
  • Thanks a lot for the version-tip. I was constantly overlooking this part and trying with Scala 2.12.x version. – Sai Kiriti Badam Sep 07 '17 at 07:11
  • Did not resolve the issue , My SBT file is : name := "HeadSpace1" version := "0.1" //scalaVersion := "2.11.11" mainClass := Some("Main") /* libraryDependencies += "org.apache.spark" %% "spark-core" % "2.3.1"*/ scalaVersion := "2.11.8" libraryDependencies ++= Seq( "org.apache.spark" %% "spark-core" % "2.1.0", "org.apache.spark" %% "spark-sql" % "2.1.0" ) – Umair Iqbal Aug 15 '18 at 12:04
2

It is about 5 years since the previous answer, but I had the same issue and the answer mentioned here did not work. So, hopefully this answer works for those who find themselves in the same position I was in.

I was able to run my scala program from sbt shell, but it was not working in Intellij. This is what I did to fix the issue:

  1. Imported the build.sbt file as a project.

File -> Open -> select build.sbt -> choose the "project" option.

  1. Install the sbt plugin and reload Intellij.

File -> settings -> Plugins -> search and install sbt.

  1. Run sbt.

Click "View" -> Tool Windows -> sbt. Click on the refresh button in the SBT window.

Project should load successfully. Rebuild the project.

  1. Select your file and click "Run". It should ideally work.
Sarvavyapi
  • 810
  • 3
  • 23
  • 35