0

I have a situation where, I need to certain functionality that is available in Spark library version 1.1.0, But I have two different platforms I need to run this application one. One uses Spark 1.1.0 and the other uses Spark 0.9.1. The functionality available in Spark 1.1.0 is not available in Spark 0.9.1.

That said, is it possible to have some compiler flags in the scala code, so that when compiling with Spark 1.1.0 certain code gets compiled and when compiling using the Spark 0.9.1. library another piece of code gets compiled?

like so :

#ifSpark1.1.0
val docIdtoSeq: RDD[(String, Long)] = listOfDocIds.zipWithIndex()
#endifSpark1.1.0

#ifSpark0.9.1
    val docIdtoSeq: RDD[(String, Long)] = listOfDocIds.mapPartitionsWithIndex{case(partId,it) => it.zipWithIndex.map{case(el,ind) => (el,ind+partId*constantLong)}}
#endifSpark0.9.1

Many thanks

0__
  • 66,707
  • 21
  • 171
  • 266
Ramdev
  • 375
  • 1
  • 3
  • 12
  • I highly recommend using Spark 1.0 and higher for new projects (unless that's not an option due to external constraints); Spark offers strong binary compatibility guarantees for its public APIs starting with Spark 1.0, so 1.0 to 1.1.0 etc. won't have these issues. – Josh Rosen Oct 21 '14 at 05:36

1 Answers1

4

There are several options.

  1. Since the two Spark versions are obviously not binary compatible, you would anyway need to provide two artifacts of your projects. Create a simple common API layer and then add two thin sub-projects in a multi-project sbt build that provide that layer for either Spark version.
  2. Use sbt-buildinfo to generate compile-time symbols for your Spark version, then use a macro method that pastes the two different types of method invocations above, depending on the Spark version.
  3. Use runtime reflection
0__
  • 66,707
  • 21
  • 171
  • 266
  • how does Solution 2, work within maven ? Is sbt-buildinfo something that is controlled globally or is it per project ? (Is it something I can add to the scala compiler/build plugin in maven ?) – Ramdev Oct 22 '14 at 14:08
  • sbt-buildinfo as the name suggests, is a plugin for sbt. It's just an example of how you can make the build tool provide you information about your libraries in generated code. You could as well use an environment variable or property. I don't know much about Maven, but I'm sure you can pass properties to the compiler so they are available in a macro. – 0__ Oct 22 '14 at 20:08