Proper structure of functional/acceptance tests

Question

I'm currently working on building an automated functional/acceptance test suite for a project, but I don't have a lot of experience writing these types of tests, so I wanted to get some input on properly structuring them. Specifically, I'm working with Arquillian's Graphene extension.

For example, say I have 3 tests, A, B, and C.

TestA : Tests logging into an account in the application. So, if test is successful, the browser should be on the account's home/info page.

TestB : Tests modifying an account's password. This would require logging in to the account, then testing the password change functionality.

TestC : Tests modifying an account's email. This would again require logging in to the account, then testing the email change functionality.

If TestA fails due to a problem with the login code, obviously TestB and TestC should fail, as well, since they require being logged in to an account.

Question: Should automated functional/acceptance tests each duplicate a process that is necessary to complete whatever the test is verifying? In this case, TestB and TestC need to login to the account before doing anything else. Should each test explicitly call something like:

/* ...initial test setup code here */
LoginPage.login(username, password);
assertTrue(onCorrectAccountPage);
AccountModification.changePassword(newPassword);

Or should I use some way of mocking an account into the session that can be used by Tests B and C so that they don't fail even if TestA (the actual login test) does?

Since these are user-acceptance tests, my thought was that they should do exactly what the user would do and log in whenever necessary, but I'm not sure if this is unnecessary duplication that should be handled differently (i.e., treated like units of functionality, similar to a standard unit test) and I wanted to get feedback from someone with more experience in this area.

Thanks in advance. Hopefully my question isn't too convoluted. :)

Answer 1

I personally have done it so that each test case replicates the users action as much as possible, but cutting out places where needs be. For example, TestA: logins in, goes to the correct website, goes to it's admin system, finds a user, and deletes part of the users information (such as name), TestB: logins in, goes to the correct website, goes to it's admin system, finds a user, and attempts to delete the users entirely via a button.

TestA and TestB end up on the same page - the user detail page. So in test A, I can do it all properly, exactly how a user does it, but in test B, I go directly to the detail page for that user, as opposed to going through the correct navigation manually. Why?

Saves time, why should I re-do the navigation in test B when test A already tests that anyway?

Remember that tests should not be dependant on each other - if all three tests fail because you cannot login - that's the whole point, you cannot login, so you cannot do any of the other actions.

Think of it as a user would. Each test has their own piece of user-viewable functionality they are testing, but if you cannot login, a user cannot see any of that or do anything with the functionality anyway. If I cannot login, I cannot change my password, or my email - so logically the tests should fail in the same manner.

Answer 2

This is really a per-project question, as both are valid approaches but in different situations one should be more preferred. In a big system I prefer to run the test case from start to finish regardless of how often this repeats steps (eg. I login for every test). I believe this is what Arran has said already (+1!). The reason I usually do this is because sometimes a state carried over from a previous screen can cause an error later and that is the sort of thing automated tests are great for finding. With these however, I make sure that the test data are all valid for the lead up steps and aim for the quickest solution possible. For example, login should always have correct user and password then when checking the login was successful, either just assume it should and deal with an exception later or search for an easily identifiable element on the page instead of a complicated 'more important' one.

With that said, you can also write tests that are testing multiple requirements in a sort of functionality flow. If the flow breaks tests should be written to identify the area in which the overall task is failing. I would only recommend this for small projects, or if testing is not a priority due to a lack of resources. For example, running a test that logs on, selects an item, puts it into a cart, goes to check out, and pays would test all this functionality and allow a team to fix the overarching 'process' rather than just several, potentially disconnected, bugs. Again however, I think the first approach is more thorough but also takes more time (but is worth doing often :)).

In the fear that my answer will become too long and block-y I won't go further into this here, but there is a lot to talk about in this regard and I would suggest sitting down and drawing out what you are trying to test in the application, now and in the future. This will probably be very revealing and encourage good test structure and automated writing practices. Hope this helps and it wasn't too long :)

Answer 3

In a user acceptance test you don't want to mock, but be as as close as possible to the way an end-user would use the system.

However, the unit test mantra one assert per test can be extended to acceptance testing: In TestA your verification logic is about asserting proper login: In TestB you don't need to repeat this verification, asserting just that password modification was handled correctly.

In JUnit, one can use assumeTrue instead of assertTrue to that end. So your TestB would become:

/* ...initial test setup code here */
LoginPage.login(username, password);
assumeTrue(onCorrectAccountPage);
AccountModification.changePassword(newPassword);

Now, if assumeTrue fails, the test simply is ignored. Your TestA will still fail, though, telling you and your end user what the real problem is.