How to build Spark 1.6.1 with SBT on Windows using Hadoop profiles?

Question

How can one activate Hadoop and YARN profiles while building Spark on Windows (8-10) with SBT?

>sbt package

The above code works, but could not activate profiles with the following:

>sbt -Pyarn package

I'm asking, because mvn is exceptionally slow compared to SBT. I have experience building Spark on Linux using both SBT and Maven.

I have two questions here. 1. Why you need to rebuild spark on Windows? Why not using one of the pre-built binary versions available in Spark site? 2. In Windows, how you try to start the sbt and the build process? are you using PowerShell, Cygwin for example? — user1314742, May 16 '16 at 10:05
Answering the first question usually leads to a huge argument about "why would you do what you would like to do", so I'll just avoid that if you don't mind. I just invoke the `sbt` command from a simple terminal. — Dyin, May 17 '16 at 07:06
I would recommend the same thing mentioned in @Vitaliy Kotlyarenko , sbt is not meant to support maven profiles, so you need to use the sbt comes with the Spark source code — user1314742, May 17 '16 at 07:38

Vitalii Kotliarenko · Accepted Answer · 2016-05-16T12:10:55.000

You have to use ./build/sbt script bundled with Spark source distribution. It calls another script sbt-launch-lib.bash that does some profile-related magic:

enableProfile () {
  dlog "[enableProfile] arg = '$1'"
  maven_profiles=( "${maven_profiles[@]}" "$1" )
  export SBT_MAVEN_PROFILES="${maven_profiles[@]}"
}

On the other side, project definition SparkBuild extends PomBuild, that allows usage of Maven project (including profiles):

override val profiles = {                                                                                                              
  val profiles = Properties.envOrNone("SBT_MAVEN_PROFILES") match {                                                                    
    ...
  }                                                                                                                             
  profiles                                                                                                                             
}

So it should work if you run it like this (using Cygwin):

sh build/sbt -Pyarn package

Nevertheless, it didn't work for me out of the box because of incorrect discovery of path to sbt-launch-lib.bash. So I've replaced in build\sbt one line:

. "$(dirname "$(realpath "$0")")"/sbt-launch-lib.bash

to

. "$(dirname "$(realpath "$0")")"/build/sbt-launch-lib.bash

How to build Spark 1.6.1 with SBT on Windows using Hadoop profiles?

1 Answers1