Using Selenium WebDriver with TestNG and DataProviders - how to assert?

Question

I'm trying to test the login functionality of a website.

I want my first test case to provide invalid credentials with the first set of data that I defined in the DataProvider method. Than I want to execute the first assert [Assert.assertTrue(errorMessage.isDisplayed());], and verify that I get an error message.

In the second test case I want to provide valid credentials with the second set of data that I defined in the DataProvider method. Than I want to execute the second assert [Assert.assertTrue(userSettings.isDisplayed());], and verify that the user settings icon is displayed.

Since the test method runs twice and both assert statements are running one test always passes and one test always fails.

How do I use assert in this scenario?

Here's the code:

package testClasses;

import java.util.concurrent.TimeUnit;

import org.openqa.selenium.By;
import org.openqa.selenium.WebDriver;
import org.openqa.selenium.WebElement;
import org.openqa.selenium.chrome.ChromeDriver;
import org.testng.Assert;
import org.testng.annotations.AfterClass;
import org.testng.annotations.AfterMethod;
import org.testng.annotations.BeforeClass;
import org.testng.annotations.DataProvider;
import org.testng.annotations.Test;

public class LetsKodeItLogin {

    WebDriver driver;

    @DataProvider(name="login")
    public Object[][] getData() {
        return new Object[][] {
            {"Oren@email.com", "test"},
            {"test@email.com", "abcabc"}
        };
    }

    @BeforeClass
    public void setUp() {
        driver = new ChromeDriver();
        driver.manage().window().maximize();
        driver.manage().timeouts().implicitlyWait(2, TimeUnit.SECONDS);
        driver.get("https://letskodeit.teachable.com");
    }

    @Test(dataProvider="login")
    public void letsKodeIt(String usernameEmail, String password) throws InterruptedException {
        WebElement loginLink = driver.findElement(By.xpath("//a[contains(@href,'/sign_in')]"));
        loginLink.click();
        WebElement emailField = driver.findElement(By.id("user_email"));
        emailField.sendKeys(usernameEmail);
        WebElement passwordField = driver.findElement(By.id("user_password"));
        passwordField.sendKeys(password);
        WebElement loginButton = driver.findElement(By.name("commit"));
        loginButton.click();

        WebElement errorMessage = driver.findElement(By.xpath("//div[contains(text(),'Invalid email or password')]"));
        WebElement userSettings = driver.findElement(By.xpath("//div[@id='navbar']//img[@alt='test@email.com']"));

//      Assert.assertTrue(errorMessage.isDisplayed());      
//      Assert.assertTrue(userSettings.isDisplayed());

    }

    @AfterMethod
    public void afterEachMethod() throws InterruptedException {
        Thread.sleep(1000);
    }

    @AfterClass
    public void tearDown() {
        driver.quit();
    }
}

Shouldn't two distinct Asserts be handled from 2 different `Test` annotations? Though there is a way out. What is your exact requirement? How would you like to mark this `Testcase` **Pass** or **Fail**? — undetected Selenium, Jan 24 '18 at 16:21
Thanks for the quick response. I’m new to TestNg, hence the question. In both asserts the test should pass. After the first set of data (Oren@email.com, test), the test should assert based on the errorMessage webelement. Second assert based on the webelement userSettings for the second set of data (test@email.com, abcabc) — oren kostin, Jan 24 '18 at 16:32
You can also pass a flag along with other inputs and based on that you can decide the assertion. — Pradeep hebbar, Jan 24 '18 at 17:41
Both your asserts are [`False Positive`](https://en.wikipedia.org/wiki/False_positives_and_false_negatives) . They would fail on successful logins. How do you want to handle it? — undetected Selenium, Jan 24 '18 at 20:08
@DebanjanB Neither of them are false positives. Only one is intended to be used at a time, the first for bad login and the second for good login. The way he's got it set up is fine, he just needs to separate the test cases so the right one gets used with the right scenario. — JeffC, Jan 24 '18 at 20:19
@JeffC Please stop jumping into conclusions so early. The question itself is way too broad and have at-least 8 use-cases to be handled. Just by saying **got it set up is fine** you are passing the buck to the OP which is misleading. I am still not sure if you have understood `False Positive` or not but on either of the _`driver.findElement()`_ will return **NoSuchElement** exception and asserts won't get triggered at all. — undetected Selenium, Jan 24 '18 at 20:36
@DebanjanB If you think it's too broad, then vote close it as broad. He's asking a simple question... how to use (basically) the same code but different inputs (logins) and assert each case. This is clearly stated at the top of his question. He doesn't intend to use them both at the same time, he's asking how to separate them. If a NoSuchElement exception is thrown, the test will fail due to the exception. If the asserts aren't run, how can they be a false positive or negative? — JeffC, Jan 24 '18 at 20:59

Julian · Answer 1 · 2018-01-24T17:43:36.347

Data providers are intended to provide an easy way to run a parameterized test with multiple datasets in an identical way. It looks like you're trying to combine two tests as one. It would be better to split them up so your generated test results have clear reporting on the specific scenario being tested. Additionally, by keeping tests focused, you're able to assert for not only the presence of stuff, but also the absence of other stuff (such as the error message in a successful scenario)

@Test
public void testValidLogin() throws InterruptedException {
    String usernameEmail = "valid@email.com";
    String password = "abcdsecurepass";
    WebElement loginLink = driver.findElement(By.xpath("//a[contains(@href,'/sign_in')]"));
    loginLink.click();
    WebElement emailField = driver.findElement(By.id("user_email"));
    emailField.sendKeys(usernameEmail);
    WebElement passwordField = driver.findElement(By.id("user_password"));
    passwordField.sendKeys(password);
    WebElement loginButton = driver.findElement(By.name("commit"));
    loginButton.click();

    WebElement errorMessage = driver.findElement(By.xpath("//div[contains(text(),'Invalid email or password')]"));
    WebElement userSettings = driver.findElement(By.xpath("//div[@id='navbar']//img[@alt='test@email.com']"));

    Assert.assertFalse(errorMessage.isDisplayed());      
    Assert.assertTrue(userSettings.isDisplayed());
}

@Test
public void testInvalidLogin() throws InterruptedException {
    String usernameEmail = "invalid@email.com";
    String password = "hackme";
    WebElement loginLink = driver.findElement(By.xpath("//a[contains(@href,'/sign_in')]"));
    loginLink.click();
    WebElement emailField = driver.findElement(By.id("user_email"));
    emailField.sendKeys(usernameEmail);
    WebElement passwordField = driver.findElement(By.id("user_password"));
    passwordField.sendKeys(password);
    WebElement loginButton = driver.findElement(By.name("commit"));
    loginButton.click();

    WebElement errorMessage = driver.findElement(By.xpath("//div[contains(text(),'Invalid email or password')]"));
    WebElement userSettings = driver.findElement(By.xpath("//div[@id='navbar']//img[@alt='test@email.com']"));

    Assert.assertTrue(errorMessage.isDisplayed());      
    Assert.assertFalse(userSettings.isDisplayed());

}

You could go further by taking the common piece of logging in and putting it in a helper method:

public void performLogin(String username, String password) throws InterruptedException {
    WebElement loginLink = 
    driver.findElement(By.xpath("//a[contains(@href,'/sign_in')]"));
    loginLink.click();
    WebElement emailField = driver.findElement(By.id("user_email"));
    emailField.sendKeys(username);
    WebElement passwordField = 
    driver.findElement(By.id("user_password"));
    passwordField.sendKeys(password);
    WebElement loginButton = driver.findElement(By.name("commit"));
    loginButton.click();
}

@Test
public void testValidLogin() throws InterruptedException {
    performLogin("valid@email.com", "abcdsecurepass"); // All the repetition we saved
    WebElement errorMessage = driver.findElement(By.xpath("//div[contains(text(),'Invalid email or password')]"));
    WebElement userSettings = driver.findElement(By.xpath("//div[@id='navbar']//img[@alt='test@email.com']"));

    Assert.assertFalse(errorMessage.isDisplayed());      
    Assert.assertTrue(userSettings.isDisplayed());
}

@Test
public void testInvalidLogin() throws InterruptedException {
    performLogin("invalid@email.com", "hackme"); // All the repetition we saved
    WebElement errorMessage = driver.findElement(By.xpath("//div[contains(text(),'Invalid email or password')]"));
    WebElement userSettings = driver.findElement(By.xpath("//div[@id='navbar']//img[@alt='test@email.com']"));

    Assert.assertTrue(errorMessage.isDisplayed());      
    Assert.assertFalse(userSettings.isDisplayed());
}

Edit: Let's say you did really want to use a data provider, you certainly could make a solution that uses it. One way could be by adding the expected success state as another parameter:

@DataProvider(name="login")
public Object[][] getData() {
    return new Object[][] {
        // email,         password, should pass?
        {"Oren@email.com", "test", true},
        {"test@email.com", "abcabc", false}
    };
}

@Test(dataProvider="login")
public void letsKodeIt(String usernameEmail, String password, boolean shouldSucceed) throws InterruptedException {
    WebElement loginLink = driver.findElement(By.xpath("//a[contains(@href,'/sign_in')]"));
    loginLink.click();
    WebElement emailField = driver.findElement(By.id("user_email"));
    emailField.sendKeys(usernameEmail);
    WebElement passwordField = driver.findElement(By.id("user_password"));
    passwordField.sendKeys(password);
    WebElement loginButton = driver.findElement(By.name("commit"));
    loginButton.click();

    WebElement errorMessage = driver.findElement(By.xpath("//div[contains(text(),'Invalid email or password')]"));
    WebElement userSettings = driver.findElement(By.xpath("//div[@id='navbar']//img[@alt='test@email.com']"));

    if (shouldSucceed) {
        Assert.assertFalse(errorMessage.isDisplayed());      
        Assert.assertTrue(userSettings.isDisplayed());
    } else {
        Assert.assertTrue(errorMessage.isDisplayed());      
        Assert.assertFalse(userSettings.isDisplayed());

}

I would argue that this isn't really a super great use of data providers, this example would likely make reading test results more difficult, because at a glance you likely wouldn't be able to know if a failure was because of a bad user passing, or a good user failing.

Thanks for the thorough response. I was confused regarding the usage of data providers. I think I now understand better their usage. In a scenario where we want to test the number of characters when entering a user name (e.g. 8-13), that will be a good use of data providers, right? — oren kostin, Jan 24 '18 at 17:29
I updated the answer to talk a bit more about data providers and show a way you could leverage them (with some caveats). You're on the right track. If you have a string, and the page shows you a counter showing the length of it. You may want to compare the actual length with the length reported. This would be a great use because your data provider could call the same method with the min length, max length... etc . It's worth saying though, that you still want to think about what the test is testing and keep it focused. Testing invalid lengths should be separate from testing valid lengths — Julian, Jan 24 '18 at 17:40
In your data provider example the test fails because when it tries to execute: WebElement userSettings = driver.findElement(By.xpath("//div[@id='navbar']//img[@alt='test@email.com']")); we get NoSuchElementException. The user settings button is not present yet. — oren kostin, Jan 24 '18 at 17:55
@orenkostin Unfortunately, I couldn't test my code. Take it with a grain of salt, I don't know the site you're trying to run this against so I don't know how it would work. I simply took the code you provided originally and shuffled it around to demonstrate patterns — Julian, Jan 24 '18 at 18:10

score 0 · Answer 2 · answered Jan 24 '18 at 17:48

You can add one more parameters to input like below. "False" flag can be used to identify negative test case and "true" flag for positive test case.

@DataProvider(name="login")
    public Object[][] getData() {
        return new Object[][] {
            {"Oren@email.com", "test",false},
            {"test@email.com", "abcabc",true}
        };
    }

Then assert the error message based on the flag , like below. (Inside test)

 @Test(dataProvider="login")
 public void letsKodeIt(String usernameEmail, String password,boolean flag) throws InterruptedException {
  ...
  ...
  ...
  if(flag){
      Assert.assertTrue(!errorMessage.isDisplayed());
  }else{
      Assert.assertTrue(errorMessage.isDisplayed());
  }

Hope this will solve your problem.

score -1 · Answer 3 · answered Jan 24 '18 at 20:13

-1

I don't think this is a good use of data providers. I would not use them in this case and would write two separate tests, one for the bad login and another for the good login. The bad login case would contain only the first assert, Assert.assertTrue(errorMessage.isDisplayed()); and the good login case would contain only the second assert, Assert.assertTrue(userSettings.isDisplayed());. Separating the two test cases makes them more readable/understandable.

answered Jan 24 '18 at 20:13

JeffC

22,180
5
32
55

I don't think think checking `valid` and `invalid` logins should be just only be in `two separate tests`but separated in two `groups`. But given OP's simple structure and logic your answer doesn't attempts to answer OP's Question anyway. But it is definitely achievable. – undetected Selenium Jan 24 '18 at 20:46
If you have a different opinion, groups vs tests, then post an answer. A difference of opinion doesn't warrant a downvote. Downvotes are for answers that are not useful (see the tooltip). I did answer the question. He asked how to use different logins with the same code and have two asserts for the different cases. I explained in my answer that he shouldn't combine the two tests and further explained which assert should go into which test. – JeffC Jan 24 '18 at 21:08
1

@JeffC, OP asking for solution not for suggestion. `I don't think this is a good use of data providers.` you are not using the same from your concern and your requirement. but OP clearly mentioned the code and asking the solution for the same. – NarendraR Jan 27 '18 at 04:16
@NarendraR Yes, and my solution is to split the tests. There's not always one way to do things. I don't think the approach OP was taking was the best one. I wasn't the only one to "suggest" that OP's approach wasn't a good idea and yet you only downvoted mine? Why is that? – JeffC Jan 27 '18 at 05:34

Using Selenium WebDriver with TestNG and DataProviders - how to assert?

3 Answers3