Outside-in BDD (with Specflow)

Question

I'm new to BDD, but I found it very interesting and want to develop my next project using BDD. After googling and watching screencasts I still have lots of questions about BDD in real life.

1. Declarative or Imperative scenarios?

Most of given-when-then scenarios I saw were written in terms of UI (imperative).

Scenario: Login
     Given I am on the Login-page
     When I enter 'AUser' in the textbox 'UserName'
       And I enter 'APassword' in the textbox 'Password'
       And I click the 'Login' button
     Then I should see the following text 'You are logged in'

I found those tests extremely brittle and they tell nothing about business value of clicking on buttons. I think its nightmare to maintain. Why most of examples use imperative scenarios?

Scenario: Login (declarative)
     Given I am not logged in
     When I log in using valid credentials
     Then I should be logged in

If you prefer declarative style, how do you describe such stuff like 'Home page' or 'Products page'?

• Tips for writing good specifications

2. Exercise UI or not?

Most of steps implementations I saw used WatiN, White or something like that to implement scenarios from user point of view. Starting browser, clicking buttons. I think its extremely slow and brittle. Well, I can use something like Page object to make tests less brittle. But thats another amount of work. Especially for desktop applications with complex UI.

How do you implement scenarios in real-life projects - exercising UI, or via testing controllers/presenters?

• Best way to apply BDD

3. Real database or not?

When Given part of scenario is implemented, often it needs some data to be in the system (e.g. some products for shop application). How do you implement that part - adding data to real database (full end-to-end testing), or providing repository stubs to controllers?

Waiting for experienced answers!

UPDATE: Added useful links on questions.

Answer 1

1. Declaritive is the proper way, IMO. If youre talking about page .aspx file names, you're doing it wrong. The purpose of the story is to facilitate communication between developers and non-develoeprs. Non-developers don't care about products.aspx, they care about a product listing. Your system does something the non-developers find value in. This is what you're trying to capture.

2. Well, the stories tell the high level features you need to implement. Its what your system must do. The only way to really tell if you've done this is to in fact exercise the UI. BDD SpecFlow stories to me don't replace unit tests, rather they're you're integration tests. If you break this, you've broken the value the business gets from your software. Unit tests are implementation details your users don't care about, and they only test each piece in isolation. That can't tell you if A and B actually work together all the time (in theory it should, in practice interesting [read: unexpected] things happen when you actually have the two parts playing with each other). Automated end to end tests can help with your QA as well. If a functional area breaks, you know about it, and they can spend their time in other areas of the application while you determine what broke the integration tests.

3. This is a tricky one. We've done a hybrid approach. We do use the database (integrate tests after all test the system functioning as one thing, rather than the individual components), but rather than resetting configurations all the time we use Deleporter to replace our repositories with Moqs when we need to. Seems to work ok, but there are certainly pros and cons either way. I think we're still largely figuring this out ourselves.

Answer 2

Edit: I found this article just now describing the concept of limiting yourself to talking only about specific domains to avoid brittle scenarios.

His main point is that the minimum number of domains you can talk about are the problem domain, and the solution domain. If you're talking about anything outside those two domains then you involve too many stakeholders, you introduce too much noise, and you make your scenarios brittle.

He also mentions an absolute "declarative" or "imperative" model being a myth. He talks about a cognative model called "chunking", saying that at any level of your abstraction, you can "chunk up" or "chunk down". This means you can get more explicit (how?) or more meta (what or why?). You chunk up from an imperative model by asking "what are we doing?" You chunk down by saying "how will we do this?" So I guess I wouldn't get too hung up on declarative vs imperative - it won't get you anywhere as far as this problem goes.

What will get you somewhere is figuring out which domains each term belongs in, possibly by identifying which stakeholder is the expert for the domain that term belongs in. Once you've identified all the domains, you can either pick related terms that are in one of the scenario's most prominent domains, or remove non-fitting statements entirely. If that isn't sufficient, you can split up, further specify, or move the scenario so it can satisfy these requirements.

BTW, he also uses the scenario of logging in on a UI, so you've got concrete guidance :)

Before Edit: (some of this still applies. The "DB or no DB" and "UI or no UI" questions are unrelated)

1 - Declarative or Imperative scenarios?

Declarative when you can, though imperative has some value (at some points in a project lifecycle).

Imperative is an easier way to think for testers and business analysts who aren't as familiar with information theory and design. It is also easier to think about earlier on in a project, before you've nailed down your problem domain and workflows. It can be useful for exploratory thinking.

Declarative is less subject to change over time. Since a GUI is the part of an application most subject to churn at a whim, this is extremely valuable. It is easier to think about once you've nailed down your problem domain and workflows, and are more focused on relational concepts. It is a more stable and more generally applicable model.

If you write test cases with a generic and declarative model, you could implement them using any combination of full app GUI automation, integration tests, or unit tests.

how do you describe such stuff like 'Home page' or 'Products page'?

I'm not sure I would at the base level of features and requirements. You might make sub-features and sub-requirements that describe implementation details, like specific UI workflows. If you're describing a piece of a UI, then you should be defining a UI feature/requirement.

2 - Exercise UI or not?

Both.

I think its extremely slow and brittle

Yes, it is. Perform every high level scenario/requirement with the UI and full DB integration, but don't exercise every single code path with end to end UI automation, and certainly not edge cases. If you do, you'll spend more time getting them working, and a lot less time actually testing your application.

You can architecture your application so you can do lower cost integration tests, including single-piece UI based tests. Unit tests are also valuable.

But the fewer integration tests you do, the more forehead-slapping bugs you're going to miss. It may be easier to write unit tests, and they will certainly be less brittle, but you'll be testing less of your application, by definition.

3 - Real database or not?

Both.

High level end-to-end integration tests must be done with the full system in place. This includes a real DB, running your tests with each system on a different server, etc.

The lower level you get, the more I advocate mock objects.

• Unit tests should only test individual classes
• Mid-level integration tests should avoid expensive, brittle, and impactful dependencies such as the file system, databases, the network, etc. Try to test the implementation of those brittle and impactful dependencies with unit tests and end-to-end tests only.

Answer 3

Instead of mentioning a page by name, describe what it represents, e.g.

Scenario: Customers logs in successfully

  When I log in
  Then I should see an overview of ACME's top selling products

You can test directly against underlying APIs or models, but the more you do this, the more you risk not catching an integration issue. One approach is to balance things with a small number of full-stack tests, and a larger number which test between two layers only.