Executing task with custom and plugin's configs in sbt?

Question

I have an SBT project that has a full build definition with a custom config used for testing.

I add a plugin that comes with its own configuration, and I want to be able to execute a task using both configurations if possible.

If not possible, I would like to pull in the configuration from the plugin into my custom config.

The specifics

My custom config is unit which is used for classifying certain tests as unit tests, and I would like to add in coverage using sbt-scoverage.

My full build definition includes:

lazy val UnitTest = config("unit") extend(Test)
def unitTestFilter(name: String) = name endsWith "UnitSpec"

lazy val myProject = Project(id="MyProject")
         .configs( UnitTest )
         .settings( inConfig(UnitTest)(Defaults.testTasks) : _*)
         .settings(
            testOptions in UnitTest := Seq(Tests.Filter(unitTestFilter)),
            parallelExecution in UnitTest := true
         )

In order to execute a unit test, I run sbt unit:test.

How would I A) add in sbt-scoverage to the unit configuration in the build definition, or B) include both scoverage and unit configurations in an SBT run, e.g. sbt "unit,scoverage:test"?

edit Based on the 2 answers provided, I am getting close to what I want. The problem I have now is with test filters, and applying it to the config. I have updated my original code snippet above to include the filter I used based on filename - see testOptions in the project section and how i used unitTestFilter. In order to add coverage, I have added new configs, for Coverage tests, so now have UnitTest, ComponentTest, CoverageUnitTest and CoverageComponentTest. The issue I have now is that I cannot get the test options / filename filters to propagate through to the scoverage config.

Method 1: creating a config that extends scoverageTest

lazy val CoverageUnitTest = config("coverUnit") extend (scoverageTest)
def unitTestFilter(name: String) = name endsWith "UnitSpec"

lazy val myProject = Project(id="MyProject")
     .configs( UnitTest, CoverageUnitTest )
     .settings(
        testOptions in CoverageUnitTest := Seq(Tests.Filter(unitTestFilter)),
        parallelExecution in CoverageUnitTest := true
     )

In the above method, the settings are not applied. If I run coverUnit:test, it executes both component and unit specs. I have also tried using testOptions in scoverageTest, which works with the filter, but the issue is that I have 2 configs in my project, namely CoverageUnitTest and CoverageComponentTest, and if I use in scoverageTest, then only 1 filter is applied to both tasks (i.e. coverUnit:test and coverComponent:test both execute unit tests, or component tests, based on order).

Method 2: extending test and adding in the instrumentation

lazy val CoverageUnitTest = config("coverUnit") extend (Test)
def unitTestFilter(name: String) = name endsWith "UnitSpec"

lazy val myProject = Project(id="MyProject")
     .configs( UnitTest, CoverageUnitTest )
     .settings( inConfig(CoverageUnitTest)(ScoverageSbtPlugin.instrumentSettings) : _* )
     .settings(
        testOptions in CoverageUnitTest := Seq(Tests.Filter(unitTestFilter)),
        parallelExecution in CoverageUnitTest := true
     )

Again, this method doesn't filter the tests with unitTestFilter.

From looking at the ScoverageSbtPlugin source, it looks like I might need to somehow look at overriding this: inConfig(scoverageTest)(Defaults.testSettings) ++, is this the right approach?

edit #2 Here's a work-around, using Method 1, with "CoverageUnitTest" replaced with "scoverageTest":

lazy val dynamicTestFilter(name:String):Boolean = name endsWith scala.util.Properties.envOrElse("TEST_TYPE","UnitSpec")

run with TEST_TYPE=ComponentSpec sbt coverComponent:test

Although I responded I'm unsure about it still. What do you mean by *"add in sbt-scoverage to the unit configuration in the build definition"*? How are you going to verify whether your build configuration meet the requirement? — Jacek Laskowski, Mar 29 '14 at 14:57
Basically want to run `unit:test` and have scoverage reports from that. I don't mind creating a new configuration, e.g. unitscoverage, that will allow me to run `sbt unitscoverage:test` to run my unit tests with coverage reports. — Brett, Mar 31 '14 at 11:32

score 1 · Answer 1 · answered Mar 29 '14 at 14:55

I use sbt 0.13.5-M2:

> about
[info] This is sbt 0.13.5-M2
[info] The current project is {file:/Users/jacek/sandbox/so/sbt-scoverage-custom-config/}myProject 0.1-SNAPSHOT
[info] The current project is built against Scala 2.10.4
[info] Available Plugins: sbt.plugins.IvyPlugin, sbt.plugins.JvmPlugin, sbt.plugins.CorePlugin, ScoverageSbtPlugin, scoverage.ScoverageSbtPlugin
[info] sbt, sbt plugins, and build definitions are using Scala 2.10.4

Unless I'm mistaken, you're asking about applying scoverage settings to unit so executing unit's test would be identical to executing test:test plus scoverage settings. If so, read on. Otherwise, please advise.

I think, you should have the following in the build definition (I picked build.sbt which was simpler for me):

scalaVersion := "2.10.4"

lazy val UnitTest = config("unit") extend(Test)

lazy val myProject = project
  .in(file("."))
  .configs(UnitTest)
  .settings(
    inConfig(UnitTest)(ScoverageSbtPlugin.instrumentSettings): _*
  )
  .settings(
    parallelExecution in UnitTest := true
  )

As I know close to nothing about sbt-scoverage I can't confirm its proper configuration other than verifying the settings:

> unit:libraryDependencies
[info] List(org.scala-lang:scala-library:2.10.4, org.scoverage:scalac-scoverage-plugin:0.97.0:scoverage)
> test:libraryDependencies
[info] List(org.scala-lang:scala-library:2.10.4)

Thanks Jacek, I have given it a run, and it's almost there - adding in the `ScoverageSbtPlugin.instrumentSettings`. however, I've updated my original post to indicate another problem I'm having. I need to filter what tests to run based on filename. Method 2 above covers this approach, can you see an easy way to apply the filter? — Brett, Apr 14 '14 at 14:57

Eugene Yokota · Answer 2 · 2014-03-29T22:36:32.420

I didn't check this but here's something you could try.

lazy val UnitTest = config("unit") extend(Test)
lazy val myProject = Project(id="MyProject").
  configs(UnitTest, scoverage, scoverageTest).
  settings(ScoverageSbtPlugin.instrumentSettings: _*).
  settings(
    parallelExecution in UnitTest := true,
    sources in scoverageTest += (sources in UnitTest).value,
    sourceDirectory in scoverageTest += (sourceDirectory in UnitTest).value,
    unmanagedResources in scoverageTest += (unmanagedResources in UnitTest),
    resourceDirectory in scoverageTest += (resourceDirectory in UnitTest),
    externalDependencyClasspath in scoverageTest := Classpaths
      .concat((externalDependencyClasspath in scoverageTest).value, (externalDependencyClasspath in UnitTest).value)
  )

Hi Eugene, thanks for the response. I tried this approach - as per Jacek's response, it was enough to use the `instrumentSettings` in my config, without needing to do all the `scoverageTest` stuff. However, I'm battling to get a test filter working. Is there anything overly obvious about my approach that I'm missing in order to filter tests? — Brett, Apr 14 '14 at 14:59

score 0 · Answer 3 · answered May 09 '14 at 13:16

I used a combination of both suggestions to come up with a working config. There's 1 ugly hack I used because I wasn't quite sure how to stop config inheritence, but here's more or less what I had (I'm only showing 1 test type - unit - here, but I actually have a number of different types. Only unit can run in parallel, component / integration test configs need parallel set to false):

lazy val UnitTest = config("unit") extend(Test)
lazy val CoverageUnitTest = config("coverUnit") extend (scoverageTest)
def unitTestFilter(name: String) = name endsWith "UnitSpec"
def dynamicTestFilter(name:String): Boolean = name endsWith scala.util.Properties.envOrElse("TEST_TYPE","UnitSpec")

def determineParallel:Boolean = {
  scala.util.Properties.envOrElse("TEST_TYPE","UnitSpec") match {
    case testType:String if testType == "UnitSpec => true
    case _ => false
  }
}

lazy val myProject = Project(id="MyProject")
 .configs( UnitTest, CoverageUnitTest )
 .configs( ComponentTest, CoverageComponentTest )
 .settings( inConfig(UnitTest)(Defaults.testTasks) : _*)
 .settings(
   testOptions in UnitTest := Seq(Tests.Filter(unitTestFilter)),
   parallelExecution in UnitTest := true
 )
 .settings( inConfig(ComponentTest)(Defaults.testTasks) : _*)
 .settings(
   testOptions in ComponentTest := Seq(Tests.Filter(componentTestFilter)),
   parallelExecution in ComponentTest := false
 )
 .settings(
    testOptions in scoverageTest := Seq(Tests.Filter(unitTestFilter)),
    parallelExecution in scoverageTest := determineParallel
 )

The first group of settings control unit tests (no coverage), executed with unit:test and the 2nd controls component tests (component:test). The last configuration is used for running tests with coverage enabled. The scoverageTest configuration is used for both unit and component test, but I set a system property to use a flag when running SBT to enable / disable parallel execution.

sbt coverUnit:test will execute unit tests using the scoverageTest configuration and enable parallel. $ TEST_TYPE=ComponentSpec sbt coverComponent:test executes the component tests in the scoverageTest configuration and disables parallel execution through the determineParallel function.

Having to use an environment variable isn't elegant at all, but worked for my use case. — Brett, May 09 '14 at 13:17

Executing task with custom and plugin's configs in sbt?

The specifics

Method 1: creating a config that extends scoverageTest

Method 2: extending test and adding in the instrumentation

3 Answers3