2

I have a bunch of integration tests running by sbt, given test N suites with 1..M tests per each suite. I have set fork in IntegrationTest := true, but test suites are always executed sequentially. According to the docs, this must not be the case: test suites should be executed concurrently.

the test suites are a class as following:

class MyTestSuite1 extends FlatSpec with Matchers 
...
it should "do A" {}
it should "do B" {}

class MyTestSuite2 extends FlatSpec with Matchers 
...
it should "do C" {}
it should "do D" {}

the problem

MyTestSuite1 and MyTestSuiteN are executed sequentially (by the alphabet order to be exact)

expectation

MyTestSuite1 and MyTestSuiteM are executed concurrently

env

.sbopts:

-J-Xms1G
-J-Xmx4G
-J-XX:MaxMetaspaceSize=512m
-J-Xss4M

note

I noticed that all test are running using the same pool and thread, for example, pool-1-thread-1 for all tests.

sbt version: 1.2.8 Scala: 2.12.8 os: MacOS 10.15, Ubuntu 19.04 Scalatest ver: 3.2.0-SNAP10

Tried sbt v. 1.3.2 - same result. Adding 

testOptions in IntegrationTest += Tests.Argument(TestFrameworks.ScalaTest, "-P4"),

does not help.

============

Update

fork in(IntegrationTest, test) := true works on a global level, but I have 2 projects and I want to make it work to preserve relative path to the proj.

e.g.

lazy val `p1` = Project(id = "p1", base = file("./p1"))
  .configs(IntegrationTest)
  .settings(Defaults.itSettings: _*)
  .settings(
    fork in(IntegrationTest, test) := true,
    ...)

lazy val `p2` = Project(id = "p2", base = file("./p2"))
  .configs(IntegrationTest)
  .settings(Defaults.itSettings: _*)
  .settings(
    fork in(IntegrationTest, test) := true,
    ...)

does not run tests in parallel

instead, this runs in parallel, but, obviously, the home dir is set to be "." rather than to be "./p1" or "./p2" respectively:

fork in(IntegrationTest, test) := true

lazy val `p1` = Project(id = "p1", base = file("./p1"))
  .configs(IntegrationTest)
  .settings(Defaults.itSettings: _*)
Nik Katalnikov
  • 103
  • 1
  • 7

2 Answers2

6

SOLUTION:

It appears there's testForkedParallel in IntegrationTest := true option which does exactly what I needed - it spawns new JVM per test suite.

==============

Remark:

So, the only problem is that now it spawns JVMs as many as the count of all available CPUs and I can't funnel only test concurrency:

OPTION 1 - funnels all sbt processes to be only 4 in parallel

concurrentRestrictions in Global := Seq(Tags.limitAll(4))

OPTION 2 - just does nothing (test are in the subproject)

concurrentRestrictions in Global += Tags.limit(Tags.Test, 4),
Community
  • 1
  • 1
Nik Katalnikov
  • 103
  • 1
  • 7
2

By default tests executed in forked JVM are executed sequentially. Refer the following para from sbt testing docs:

The setting:

Test / fork := true

specifies that all tests will be executed in a single external JVM. See Forking for configuring standard options for forking. By default, tests executed in a forked JVM are executed sequentially. More control over how tests are assigned to JVMs and what options to pass to those is available with testGrouping key.

So, you have two options:

  1. Do not fork JVM and your tests will run parallelly by default
  2. If you want to fork and still rut tests parallelly go through this docs: https://www.scala-sbt.org/1.x/docs/Testing.html#Forking+tests
Pritam Kadam
  • 2,388
  • 8
  • 16