How should automated acceptance tests use set up the initial state?

Question

I am developing the acceptance test for a web service. The tests are driven through Specflow and using SQL Server CE as database. The service shares the DB with other applications and modules and uses some data created through one of those apps.

From the point of view of my product, there are two types of "data":

1. Data that we only consume
2. Data that we create/modify

Before running a test case, you set up the initial state of the database. For the consumed only data, the only way to initialise the data is inserting directly in the database. But for managed data we either can set the state directly in the DB, or do it as a user would do it, calling the API.

For example, I want to test that my method 'updateItem' updates correctly the price of the item to 'y'. To be able to execute this method I need to set up my database with an initial item with price 'x'. There are two ways of doing this:

1. Set the state of the item directly in the database
2. Call 'createItem'

The pros for the first method is that I only execute the method I want to test, so if the test fails, only fails for one reason. The cons would be that if in the future the state of the created item changes (adding a new field), I need to make that change in the initial state manually.

In the other hand, with the second method the pros and cons would be the opposite. The test could fail because 'createItem' failed, although some test frameworks would tell you that it was the set up what failed, not the actual test. But whatever changes occur in 'createItem', those changes are automatically included in your test and you don´t have to update the initial state manually.

Any advice is much appreciated.

Answer 1

To be fair you've pretty much nailed your immediate issues in the question, you really just need to make your decision.

There are many ways of setting up this data, more than you have mentioned, and while this question isn't quite a duplicate of SpecFlow Integration Testing with Database Patterns, have a quick look at that first.

Ok, so now to choose, Is it better to use the existing API, or to not? Well if not, what are going to do? I'm guessing you are going to write some additional SQL statements and call those directly.

Doesn't that mean that you have created another API? You've got the external API, which sets the price to Y, and your new internal/testing API that sets the price to X. So you've doubled the code you have to write, and maintain. Worse, the chances are that the quick and dirty code for the tests will actually end up copied and pasted everywhere throughout your tests, so when it breaks you'll have to fix it in many many places.

If you can, stop thinking databases. Think APIs. You mentioned that you have some static data that just needs to be in place for the tests, so put together an API to do that, it doesn't matter if its not public, or only used by your application. Get that API in place.

Next, substitute what's behind the API. Use Mocks, use an in-memory representation, a NoSQL db, an XML file or whatever you need to make that individual test pass, because over time you are going to waste so much time to keeping that DB working. As your tests increase its going to get slower very quickly, and to be fair, you as an application developer, do not care if SQL CE is the best solution. When the system architect (even if that is you with different hat on) then comes to say we need to come of this technology and move to something else, at least you are not tied to it.

Answer 2

Alski's answer is good, but i think there is something else you should consider. Whilst calling the API of you application to create your item is fine, you should be wary of the long term implications of this approach, in that it can make your tests take significantly longer than they need to. If all you are doing is calling CreateItem, then it is ok, but imagine as your application grows in complexity, if you have to follow the users workflow all the time you might end up having to do a significant number of steps just to get the data in the state you want. If you need to test that items which are out of stock are visible on the reorder page then you might need to do all of these:

• CreateUser
• CreateProduct
• SetProductStockLevel
• CreateOrderForAllStock
• ShipOrder

Just to make sure your product exists and is out of stock. This might take significant amounts of time (especially if your tests are selenium based).

If you just inserted the correct data into the tables in the first place this would run much more quickly.

Whether this is an issue for you, I can't say, but bear this in mind before you make a decision.

Luckily specflow makes changing your mind about this quite easy. Juts have a step which is something like

Given all the stock for <Product> has been sold

then you can implement it by calling other steps to do the users workflow or pushing data directly into the DB.