0

I am having a build problem. Here is my sbt file:

name := "SparkPi"
version := "0.2.15"
scalaVersion := "2.11.8"

// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.10
libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.0.1"

// old:
//libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.1"

// https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.0.002"

scalacOptions ++= Seq("-feature")

Here is the full error message I am seeing:

[info] Set current project to SparkPi (in build file:/Users/xxx/prog/yyy/)
    [info] Updating {file:/Users/xxx/prog/yyy/}yyy...
    [info] Resolving jline#jline;2.12.1 ...
    [info] Done updating.
    [info] Compiling 2 Scala sources to /Users/xxx/prog/yyy/target/scala-2.11/classes...
    [error] /Users/xxx/prog/yyy/src/main/scala/PiSpark.scala:6: object profile is not a member of package com.amazonaws.auth
    [error] import com.amazonaws.auth.profile._
    [error]                           ^
    [error] /Users/xxx/prog/yyy/src/main/scala/PiSpark.scala:87: not found: type ProfileCredentialsProvider
    [error]     val creds = new ProfileCredentialsProvider(profile).getCredentials()
    [error]                     ^
    [error] two errors found
    [error] (compile:compileIncremental) Compilation failed
    [error] Total time: 14 s, completed Nov 3, 2016 1:43:34 PM

And here are the imports I am trying to use:

import com.amazonaws.services.s3._
import com.amazonaws.auth.profile._

How do I import com.amazonaws.auth.profile.ProfileCredentialsProvider in Scala?

EDIT

Changed sbt file so spark core version corresponds to Scala version, new contents:

name := "SparkPi"
version := "0.2.15"
scalaVersion := "2.11.8"

// https://mvnrepository.com/artifact/org.apache.spark/spark-core_2.11
libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.1"

// old:
//libraryDependencies += "org.apache.spark" %% "spark-core" % "2.0.1"

// https://mvnrepository.com/artifact/com.amazonaws/aws-java-sdk
libraryDependencies += "com.amazonaws" % "aws-java-sdk" % "1.0.002"

scalacOptions ++= Seq("-feature")
Daniel Kats
  • 5,141
  • 15
  • 65
  • 102

1 Answers1

0

You are using scalaVersion := "2.11.8" but library dependency has underscore 2.10 spark-core_2.10 which is bad.

libraryDependencies += "org.apache.spark" % "spark-core_2.10" % "2.0.1"
                                                          ^

change 2.10 to 2.11

libraryDependencies += "org.apache.spark" % "spark-core_2.11" % "2.0.1"

`

Nagarjuna Pamu
  • 14,737
  • 3
  • 22
  • 40