3

The setup

So, basically i am trying to achieve Selenium tests that run in parallel using JUnit.

For that i have found this JUnit runner. It works really well, i like it alot.

However, i am running into problems regarding the handling of WebDriver instances.

What i want

Each WebDriver element should be created once for every class before @Test methods are executed.

Logically, i could use the classes constructor for this. Actually this is quite the requirement for my tests because i need to make use of the @Parametersso that i can create the WebDriver instance accordingly (Chrome,FF,IE ...).

The problem

The problem is that i want the WebDriver instance to be cleared (driver.quit() ) after a class is done and not after each @Test method is done. But i cannot use @AfterClass because i cannot make WebDriver a static member, since each class instance has to use its own (otherwise tests would try to run in the same browser).

A possible solution

I have found a possible suggestion here by Mrunal Gosar. Following his advise i have changed WebDriver to be a static ThreadLocal<WebDriver> instead and then i create instances of it in each constructor using

 // in the classes constructor

driver = new ThreadLocal<WebDriver>() {
           @Override
           protected WebDriver initialValue() {
                 return new FirefoxDriver(); /
           }
};

Needles to say that i replaced every driver.whatever call with driver.get().whatever in my code.

Now, to address the ultimate purpose of this i also wrote a @AfterClass method that would call driver.get().quit(); which is now accepted by the compiler, as the variable is static.

Testing this however leads to unexpected behavior. I have a Selenium Grid setup with 2 nodes running on a remote machine. I had this setup running as expected before, but now browsers are spammed all over and tests fail. (While 2 browsers should be running instead 8+ are opened)

The thread i linked suggesting this solution had someone commenting that it might be a bad idea to manually handle threads if already using a framework like JUnit.

My Question

What is the right design to do this?

I could only think of

  1. Make what is suggested here work
  2. Write a single @Test annotated method that executes all other methods and then use @After to achieve the same as @AfterClass
  3. Save the constructor parameter in a member variable and deal with the fact that i have to create a browser before each @Testannotated method is executed (Using @Before to create the WebDriver instance and @After to close the session)

I don't quite know if option 3 runs into possible problems though. If i close the session after each method, then the grid-Server might actually open a new session with an entirely new class on this node before this one has finished the previous ones. While the tests are independent of each other, i still feel like this is potential danger.

Is anyone on here actively using a multithreaded Selenium test suit and can guide me what is proper design?

Community
  • 1
  • 1
Mercious
  • 378
  • 3
  • 25
  • A quick note: The problems i am experiencing with my current solution is bound to the fact that the class-constructor seems to be called for every class-method. So instead of having x amount of class instances, while x is the amount of parameters in the list returned from the `@Parameters` function, i actually have x*y where y = the amount of `@Test` annotated methods inside of my class. I dont quite know what to do about that. Maybe the Runner i am using needs to be adjusted in some way, but i doubt that i can do a better job at it. Maybe this behavior is also necessary, i don't know. – Mercious May 21 '15 at 10:19
  • I have no tried another solution involving ThreadLocal. With this, i have declared ThreadLocal sWebDriver as a static member variable and have overwritten initialValue right there. Then i declare a "normal" `WebDriver myWebDriver;` also as member variable. Then, in `@Before` i do `myWebDriver = sWevDriver,get();` and use myWebDriver in the rest of the code. This works in such that only 2 instances of WebDriver are opened and the tests use the correct instance. However, in `@AfterClass` i defines `sWebDriver.get().quit()` which seems to work only for one of the two instances. – Mercious May 21 '15 at 11:54
  • To make sure what you want to achieve: **1.** Write one class with T number of test cases (lets call it T1, T2, ...). **2.** Use JUnit parametrization (P browser types, lets call it P1, P2, ...) to make JUnit run PxT number of test cases. **3.** For particular P type set of test cases (PxT1, PxT2, PxT3, etc) instantiate specific Px WebDriver only once and destroy it at the end. **4.** Parallelize execution of test cases but only on browser type (in other words, guarantee that no test with the same Px will run concurrently). / Is that the case ? – Tomasz Domzal May 24 '15 at 11:37
  • Theoretically ... it would be possible that more instances of one browser are executed in parallel. So i need not to guarantee that a browser is only run once. HOWEVER, you bring up an interesting point and honestly, it would most likely be sufficient that 3 classes are ran in parallel (actually one class with 3 different parameters (browsers)). That would already deliver some speed and make things more sufficient. But the question remains. For those 3 instances of the class ran in parallel, how do i instance my webdriver so that i can still clear it in @AfterClass? – Mercious May 26 '15 at 07:43

1 Answers1

5

In general I agree that:

it might be a bad idea to manually handle threads if already using a framework like JUnit

But, looking at Parallelized runner you mentioned and internal implementation of @Parametrized in junit 4.12 it is possible.

Each test case is scheduled for execution. By default junit executes test cases in single thread. Parallelized extends Parametrized in that way that single threaded test scheduler is replaced with multi-thread scheduler so, to understand how this affects way Parametrized test cases are run we have to look inside JUnit Parametrized sources:

https://github.com/junit-team/junit/blob/r4.12/src/main/java/org/junit/runners/Parameterized.java#L303

Looks like:

  1. @Parametrized test case is split to group of TestWithParameters for every test parameter
  2. for every TestWithParameters instance of Runner is created and scheduled for execution (in this case Runner instance is specialized BlockJUnit4ClassRunnerWithParameters)

In effect, every @Parametrized test case generates group of test instances to run (single instance for every parameter) and every instance is scheduled independently so in our case (Parallelized and @Parametrized test with WebDriver instances as parameters) multiple independent tests will be executed in dedicated threads for every WebDriver type. And this is important because allows us to store specific WebDriver instance in scope of the current thread.

Please remember this behavior relies on internal implementation details of junit 4.12 and may change (for example see comments in RunnerScheduler).

Take I look at example below. It relies on mentioned JUnit behavior and uses ThreadLocal to store WebDriver instances shared between test in the same cases groups. Only trick with ThreadLocal is to initialize it only once (in @Before) and destroy every created instance (in @AfterClass).

package example.junit;

import java.util.ArrayList;
import java.util.Arrays;
import java.util.Collection;
import java.util.Collections;
import java.util.Iterator;
import java.util.List;

import org.apache.commons.lang3.StringUtils;
import org.junit.AfterClass;
import org.junit.Before;
import org.junit.BeforeClass;
import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.openqa.selenium.firefox.FirefoxDriver;

/**
 * Parallel Selenium WebDriver example for http://stackoverflow.com/questions/30353996/selenium-and-parallelized-junit-webdriver-instances
 * Parallelized class is like http://hwellmann.blogspot.de/2009/12/running-parameterized-junit-tests-in.html
 */
@RunWith(Parallelized.class)
public class ParallelSeleniumTest {

    /** Available driver types */
    enum WebDriverType {
        CHROME,
        FIREFOX
    }

    /** Create WebDriver instances for specified type */
    static class WebDriverFactory {
        static WebDriver create(WebDriverType type) {
            WebDriver driver;
            switch (type) {
            case FIREFOX:
                driver = new FirefoxDriver();
                break;
            case CHROME:
                driver = new ChromeDriver();
                break;
            default:
                throw new IllegalStateException();
            }
            log(driver, "created");
            return driver;
        }
    }

    // for description how to user Parametrized
    // see: https://github.com/junit-team/junit/wiki/Parameterized-tests
    @Parameterized.Parameter
    public WebDriverType currentDriverType;

    // test case naming requires junit 4.11
    @Parameterized.Parameters(name= "{0}")
    public static Collection<Object[]> driverTypes() {
        return Arrays.asList(new Object[][] {
                { WebDriverType.CHROME },
                { WebDriverType.FIREFOX }
            });
    }

    private static ThreadLocal<WebDriver> currentDriver = new ThreadLocal<WebDriver>();
    private static List<WebDriver> driversToCleanup = Collections.synchronizedList(new ArrayList<WebDriver>());

    @BeforeClass
    public static void initChromeVariables() {
        System.setProperty("webdriver.chrome.driver", "/path/to/chromedriver");
    }

    @Before
    public void driverInit() {
        if (currentDriver.get()==null) {
            WebDriver driver = WebDriverFactory.create(currentDriverType);
            driversToCleanup.add(driver);
            currentDriver.set(driver);
        }
    }

    private WebDriver getDriver() {
        return currentDriver.get();
    }

    @Test
    public void searchForChromeDriver() throws InterruptedException {
        openAndSearch(getDriver(), "chromedriver");
    }

    @Test
    public void searchForJunit() throws InterruptedException {
        openAndSearch(getDriver(), "junit");
    }

    @Test
    public void searchForStackoverflow() throws InterruptedException {
        openAndSearch(getDriver(), "stackoverflow");
    }

    private void openAndSearch(WebDriver driver, String phraseToSearch) throws InterruptedException {
        log(driver, "search for: "+phraseToSearch);
        driver.get("http://www.google.com");
        WebElement searchBox = driver.findElement(By.name("q"));
        searchBox.sendKeys(phraseToSearch);
        searchBox.submit();
        Thread.sleep(3000);
    }

    @AfterClass
    public static void driverCleanup() {
        Iterator<WebDriver> iterator = driversToCleanup.iterator();
        while (iterator.hasNext()) {
            WebDriver driver = iterator.next();
            log(driver, "about to quit");
            driver.quit();
            iterator.remove();
        }
    }

    private static void log(WebDriver driver, String message) {
        String driverShortName = StringUtils.substringAfterLast(driver.getClass().getName(), ".");
        System.out.println(String.format("%15s, %15s: %s", Thread.currentThread().getName(), driverShortName, message));
    }

}

It will open two browsers and execute three test cases in every browser window concurrently.

Console will print something like:

pool-1-thread-1,    ChromeDriver: created
pool-1-thread-1,    ChromeDriver: search for: stackoverflow
pool-1-thread-2,   FirefoxDriver: created
pool-1-thread-2,   FirefoxDriver: search for: stackoverflow
pool-1-thread-1,    ChromeDriver: search for: junit
pool-1-thread-2,   FirefoxDriver: search for: junit
pool-1-thread-1,    ChromeDriver: search for: chromedriver
pool-1-thread-2,   FirefoxDriver: search for: chromedriver
           main,    ChromeDriver: about to quit
           main,   FirefoxDriver: about to quit

You can see that drivers are created once for every worker thread and destroyed at the end.

To summarize, we need something like @BeforeParameter and @AfterParameter in context of execution thread and quick search shows that such idea is already registered as issue in Junit

Tomasz Domzal
  • 116
  • 1
  • 5
  • Thanks a lot for this post, Thomasz! I upvoted it, but allow me to ask: Why do we have to initialize the WebDriver instance in the `@Before` method, instead of the `@BeforeClass` method? Thinking of it, i guess i have misunderstood the way Parallelized worked. I thought that for every parameter in `@Parameters` there would be ONE class instance running it's tests with this parameter. Instead, it seems, that for every single test-method there is a thread running? – Mercious Jul 22 '15 at 10:40
  • I think that is was ultimately confused me. Assuming that there would be 3 class instances for 3 Parameters in the Parameter-List, i thought that it must be the most efficient way to have the driver initialized ONCE for the class, in `@BeforeClass`. I thought that it would be inefficient to instance them x times for x amount of `@Test` methods. But, if i am right regarding the thread instancing, then every class-instance only runs for a single test-method. So then there is no difference if we do `@Before` or at `@BeforeClass`. I hope i got that right? – Mercious Jul 22 '15 at 10:43
  • Code in `@BeforeClass` is executed only once and by _main_ thread (also when `@Parametrized` is used). So yes, you would iterate and initialize all required WebDriver instances but cannot store them in `ThreadLocal` to be available later in worker threads execution context. At the other hand, cleaning is possible and simpler in `@AfterClass` - it's called once and we can just iterate on all registered `WebDriver` instances. – Tomasz Domzal Jul 22 '15 at 11:21
  • Oh, that is new. Very interesting, indeed. Well, thanks a lot! I will definitely try to implement this. The whole thing has been worked on a lot since this question, but it now also uses such WebDriverFactory, so it should be easy to implement what you suggested. – Mercious Jul 22 '15 at 11:40
  • It is quite easy to extract all parallel related logic to superclass, take a look at this [gist](https://gist.github.com/tdomzal/1765066b5876f4518074). – Tomasz Domzal Jul 22 '15 at 11:54
  • Yea, that's pretty much what i did with the test already and then later someone refactored the driver generation a little. However, we can still add the ThreadLocal thing. It was hard to find a proper design for this, despite it being quite commonly applicable for Selenium. The tests gains a huge bonus from multithreading as they are heavily I/O dependent. Hope if there is someone with similar questions he finds this thread and sees your well designed example code! – Mercious Jul 22 '15 at 12:09