How can I get a consistent result on running Selenium Webdriver (Java) test cases?

Question

I want a consistent output when running Selenium WebDriver test cases.

I have more than 30 test cases in a java class. When I run it i'm getting errors in different methods. For eg. Some times it fails in the 12th test case, some times in 8th, some times in 2nd method, Some times all are running without any error. I'm sure all the test cases are correct.

It tests very fast. I thought this would be the problem. So I added

new FireFoxDriver().manage().timeouts().implicitlyWait(20, TimeUnit.SECONDS);

Even though its not consistent

Can anyone help me asap?

Answer 1

I'll disagree strongly with Arran's statement that UI testing is very unreliable. It absolutely doesn't have to be. Learning solid approaches to two main issues usually smooths out a lot of problems.

Intermittent failures are most often caused by synchronization issues around dynamic content, or locators which aren't flexible enough to handle changing conditions.

For example, you're extracting a row from a grid/table, and your locators are positional based. (3rd row, 5th cell) The sort order differs on one run, so the test breaks. Some server-side technologies like ASP.NET generate dynamic IDs which can vary, so that's another problem.

I'd encourage you to carefully evaluate your locators/selectors and see if they're robust. Avoid hardwired xpath, use IDs wherever possible--but make sure they aren't dynamic IDs and deal with them appropriately if they are.

You mentioned you've tried using an implicit wait. Make sure you understand the difference between implicit and explicit waits. (Read more here in Selenium docs.) It may be you need explicit waits for the exact conditions for the next steps in your scripts. Do not, repeat NOT use Thread.Sleep or similar manual delays in your tests except in extraordinary circumstances. WebDriver's Wait is a much better approach.

Answer 2

I find that Selenium tests often fail because page elements are not loaded properly, even if you have set implicit waits. If a test is failing inexplicably at the same point this works for me. Try this (my code is C# but you can do it in other languages):

In test initialisation set up a wait:

globalWait = new WebDriverWait(driver, TimeSpan.FromSeconds(20));

Then call the wait when a troublesome page loads: globalWait.Until(ExpectedConditions.ElementIsVisible(By.Id("myPageElement")));

This forces a wait until the element is visible. It solved a lot of problems for me. Often I set the id to a footer element, or if it is an AJAX page, one of the AJAX retrieved elements. This makes sure that the page has loaded correctly.

Another technique is to write a common routine for login and logoff, and if you are doing a lot of complex tests, logon, do some tests, logoff, then logon and do some more tests. This seems to lead to less spurious errors in Selenium.

My C# login routine:

private void Login()
{
    driver.Navigate().GoToUrl(base_url + "/login");
    globalWait.Until(ExpectedConditions.ElementIsVisible(By.Id("username")));
    driver.FindElement(By.Id("username")).SendKeys(username);
    // clear IE password autocomplete
    driver.FindElement(By.Id("password")).Clear();
    driver.FindElement(By.Id("password")).SendKeys(password);
    driver.FindElement(By.Id("btn_login")).Click();
}

My logout routine:

private void LogOut()
{
    driver.FindElement(By.Id("lnkMenuLogout")).Click();
}

A test using them:

[TestMethod]
public void EditAccount()
{
    Login();

    // navigate to account page
    driver.FindElement(By.Id("ulMenu")).FindElement(By.Id("lnkMenuAccount")).Click();
    Assert.IsTrue(driver.FindElement(By.Id("headName")).Displayed, "Player name not displayed");

    string playerChangeName = "Test" + DateTime.Now.ToString("yyyyMMddHHmmss");

    driver.FindElement(By.Id("butEditName")).Click();
    driver.FindElement(By.Id("surname")).Clear();
    driver.FindElement(By.Id("surname")).SendKeys(playerChangeName);
    driver.FindElement(By.Id("butSaveName")).Click();
    driver.Navigate().Refresh();

    // check that edit worked
    Assert.IsTrue(driver.FindElement(By.Id("headName")).Displayed, "Player name not displayed after change");
    IWebElement element = driver.FindElement(By.Id("headName"));
    string playerName = element.Text;
    StringAssert.Contains(playerName,playerChangeName,"Player name has not been changed");

    LogOut();
}

My final conclusion though is that presently, Selenium is still somewhat 'flakey'. I am currently experimenting with PhantomJS but that appears to be no better. Selenium is the best we have.

Answer 3

My best answer for you is to add "reliability and recovery code".

For instance, maybe your tests are trying to click something, but randomly the webelement is stale for whatever reason. So add code that catches StaleElementExceptions and retries a max of 5 times before failing for real.

Or, maybe sometimes the page loads a little too slowly and the content hasn't been updated before Selenium thinks its done (AJAX functionality). This causes the check to fail but when looking at a screen shot, the page is actually correct because this all happens in less than .2 seconds. So, add a selenium wait which waits for the content to be updated before using the assert call.

This essentially gives you more reliable results since your test code now knows how to handle common situations with the selenium framework.

Hope this helps!