BDD-testing using a UI driver (e.g. Selenium for a web-application)

Question

Can BDD (Behavior Driven Design) tests be implemented using a UI driver?

For example, given a web application, instead of:

• Writing tests for the back-end, and then more tests in Javascript for the front-end

Should I:

• Write the tests as Selenium macros, which simulate mouse-clicks, etc in the actual browser?

The advantages I see in doing it this way are:

• The tests are written in one language, rather than several
• They're focussed on the UI, which gets developers thinking outside-in
• They run in the real execution environment (the browser), which allows us to
  • Test different browsers
  • Test different servers
  • Get insight into real-world performance

Thoughts?

Answer 1

We've done this for a C# application using a WPF testing tool (WipFlash) and writing NUnit tests in a BDD-like fashion.

e.g.

Given.TheApplicationWindowIsOpen();
When.I.Press.OKButton();
The.Price.ShouldBeCalculated();

We had to code a lot of the DSL ourselves, needless to say. But it becomes a business/customer readable solution.

Answer 2

Try using SpecFlow with WatiN: (I'm not sure if you're using .NET here)

http://msdn.microsoft.com/en-us/magazine/gg490346.aspx

Answer 3

For web testing, you could try WebDriver. The Selenium team are busy integrating WebDriver at the moment. Simon Stewart from Google, who created WebDriver, blogged here about how it works differently to Selenium.

WebDriver uses different technologies for each browser. For Internet Explorer, WebDriver uses Microsoft's UI automation - the same technology on which WipFlash which @Brian Agnew mentioned is based. This is as close as you'll get to pretending to click buttons. Simon's blog shows why this approach can be more powerful than Selenium's Javascript solution.

WebDriver is available from the Selenium site but hasn't been fully implemented as part of Selenium yet.

Answer 4

For BDD, and any use-case driven tests, it is important to be able to communicate what a test is doing. The problem with many test suites is that post-writting nobody is quite certain exactly what the test is doing. This will come up very often if you write in a non-specialized language. Specialization doesn't necessarily mean a special language, but just enough of an abstraction in the one language so it is clear what is happening.

For example, a lot of tests have code that looks like this (pseudo-code, I won't pick on any particular framework):

object = createBrowser()
response = object.gotoURL( "http://someurl.com" );
element = response.getLink( "Click Here" );
response = element.doClick();

This is hard for somebody to quickly translate to a business driver (product manager perhaps, or user). Instead you want to create specialized functions, or a language if you're adventurous, so you can have this:

GotoURL http://someurl.com/
Click link:Click Here

Selenium, and its macros or interface, are still fairly low-level in this regards. If you do use them then at least build some wrappers around them.

You can of course also use a product called TestPlan. It has Selenium in the back-end and exposes a high-level API, and a custom langauge for testing. It also goes beyond just the web to included Email, FTP, etc. The sample language above is a TestPlan snippet

Answer 5

If this would help someone, I've written a driver builder library for that purpose: https://github.com/morsh/driver-pattern

BDD feat. Driver & Builder Patterns

Which enabled you to write your driver like this:

describe("<ToDoList />", () => {
  const driver = createDriver(Component, {
    getters: {
      todoItems: 'todo-item',
      addButton: 'add-button',
      todoTitle: (index: number) => screen.queryAllByTestId('todo-item')[index],
      deleteButton: (index: number) => screen.queryAllByTestId('delete-button')[index],
    },
    actions: get => ({
      clickAdd: () => fireEvent.click(get.addButton()),
      clickDelete: (index: number) => fireEvent.click(get.deleteButton(index)),
    }),
    createStore,
  });

  beforeEach(() => {
    drive.initialize()
  });

  it("should display list of todo items", () => {
    const todos = someTodoItems();
    driver.givenState.todos(todos).when.render();
    expect(driver.get.todoItems()).toHaveLength(todos.length);
  });

  it("should click on add", () => {
    const onClickSpy = jest.fn();
    driver.given.onClick(onClickSpy).when.render();
    expect(onClickSpy).toHaveBeenCalled();
  });
});

Answer 6

You can certainly do some of your acceptance tests this way, but I think most BDD advocates would not advise using this for all tests. And of course, true BDD advocates wouldn't call them tests...

The RSpec Book advocates a two-level cycle with acceptance tests (or Scenarios) written first (primarily in Cucumber), and unit tests written (in RSpec) in an inner cycle more resembling the traditional TDD.

The outer cycle of acceptance testing can also use tools like Selenium to drive the entire application through the UI (and the authors of The RSpec Book spend a chapter on this). But it is not appropriate for unit tests.

Tests exercising the entire application through the UI are harder to make repeatable, and have a tendency to be slower and more fragile than unit tests.

Answer 7

Actually you could do both - make a user-centric Driver interface (agnostic of GUI / tech / impl). You could then write a UIDriver and a APIDriver and choose a driver to run a specific test. Running through the UI is usually slower (out of proc, control repaints but somehow creates a higher level of confidence initially). Running through the API is much faster (in proc, easy setup-teardown).

The trick here is to separate the What from the How. Otherwise you will end up with ObscureTests and high test maintenance. Ensure the primary focus on testing and not automation.