1

We are just getting started with BDD in our company, and we are currently trying to write our first features. We have created something like the following:

Feature: There can only be one
It is important that there is only one Highlander in the system for a given time period.

Scenario: Inserting a new Higlander for a time period already taken by another 
Given there is a Highlander Adam in the system for time period 1820-1900
When I insert a Higlander Bert for time period 1800-1900
Then the System raises Error Code "ONLY1"  

Scenario: Inserting a new Higlander for a free time period
Given there is a Highlander Adam in the system for time period 1820-1900
When I insert a Higlander Bert for time period 1901 - 1902
Then the System raises no Error

My question ist about the error code (ONLY1) in the first scenario. Currently, our system raises many errors, but we do not have a simple way to identify a specific error condition. Our System ist more than 15 years old. Isn't it funny that we havn't found this to be an issue for so many years? I find it great that BDD makes it so obvious that we need a clear, shared language to identify our error messages.

Some of my colleagues argue that there is no need for an error code. This means that we have to write the assertion like this:

Then the System shows the error "There can only be 1 Highlander for a given time period."

This makes me cringe, because

  • The test will break if we later decide to change the text to something else
  • The test will only be successful on the english version of the software (we support several languages)
  • An error Code is great for finding information in a knowledge base or in suppoert tickets.

Of course, we should show the error code and a readable description of the error in the users language, but in the test spec, I would prefer to only mention the error code.

How do you test for errors? do you use codes or exceptions or just plain text?

Best Regards Mat

user1691896
  • 107
  • 7
  • 1
    I will give you my experience. When testing calculation (back end), we work with enums-error codes, because our calculation code does not have knowledge of what string is going to be written. Thus it is natural to use error codes. On the other hand, if you would like to test UI, like what string is presented, then you can use both strings and error code, depending how you configured your dependencies. At the end, you maybe can use strings for unit tests. But this is only our experience, and you need to figure out what suits you the most. – Stefan Feb 08 '20 at 13:22

1 Answers1

3

Speaking strictly from a BDD persepective, you want to stay focused on the behavior that is expected. Keep your tests devoid of technical details. That being said, error codes are technical details. When trying to adhere to BDD, avoid error codes in your feature files. Put the error codes in your step definitions instead.

User friendly error messages are a little different though. An error message shown to an end user is not a technical detail in the same sense as an error code or enum. It is something that an end user should understand. If they do not understand the user friendly error message, then it is not user friendly and should be rewritten until it is. So your example below is perfectly valid BDD:

Then the System shows the error "There can only be 1 Highlander for a given time period."

You should also understand the drawback to this approach. Any change to the message terminology means you need to update all tests that reference this message. While putting the message in your features might not violate BDD, it does make your test suite more labor intensive to maintain. A better assertion captures the essence of the behavior without requiring the scenario author to know the exact details of the user interface:

Then the system says there can only be 1 highlander

The step definition is then free to use error codes, enums or UI error messages as part of the assertion. This provides a layer of abstraction between the implementation and the description of the behavior. A functional change to the UI or code that does not affect the overall behavior can be fixed in one spot.

Check out BDD 101: Writing Good Gherkin for some good guidance.

Greg Burghardt
  • 17,900
  • 9
  • 49
  • 92
  • I agree with Greg, but i would like to point to one more thing. It is about for who do you write that automation. For your, self verification (developers/team) or because your client, product owner, project manager or somebody else (from business side) requested for it. In my team we don't have direct link with clients, and we don't have anyone who watches those automated scenarios. Therefore we have absolute freedom to name steps or scenarios however we want (what seems most appropriate to us, at given moment). – Stefan Feb 12 '20 at 15:53
  • In BDD theory, they say your gherkin specification should be behavior oriented, and i agree, but sometimes you might want just to come to good enough solution at specific moment. I think you should agree with team or whoever is involved in topic, and start with one approach, and during the time see if that works for you. – Stefan Feb 12 '20 at 15:53
  • There is nothing wrong with being pragmatic, but if your scenarios don't have any connection to your end users then I'm not sure if behavior driven development is the right tool. At that point you are equally well off just writing tests in Java, C#, Python or whatever your tech stack uses. There is nothing wrong with translating the [examples from a 3 Amigos session](https://cucumber.io/blog/bdd/example-mapping-introduction/) to straight, code-only automated tests. – Greg Burghardt Feb 12 '20 at 15:59
  • i don't think of BDD as a tool, actuallyit is an sequence of agile practices, it is a way of development, whole process how your team work on solution and deliver it to end users. Taking notes in 3 amigos session is nice, writing them for acceptance criteria and then automate them is great if team can manage it and have solid practice to do it. But automation is just part of BDD. And brings quality to software in similar way as UT, UI tests, performace or component tests do. – Stefan Feb 12 '20 at 16:15