5

I am running into a bit of a conundrum. We have an antiquated system that I am writing Behat tests for. It works great for the most part. But I have noticed an issue where the Behat tests will fail if the data I am testing against the current environment was meant for/pulled from a different environment.

For example, if I test a search by phone function in QA and expect it to return a specific entity id I cannot use that same phone number and entity id to test in RC or Live. So I would like a manageable way to maintain the testing data for each environment in Behat.

A couple thoughts have been throw around here such as putting the data into the profile (highly undesireable) or creating CSV files for each feature. I also am thinking about building all the data-specific scenarios using tables or scenario outlines and having an environment column that will be used to check against the current environment and skip when the row is not for the current environment. Maybe using a Background or some other hook to help out with this.

Does anyone know of a good way or best practice for dealing with multiple environments with different data sets in each with Behat?

Patrick
  • 3,302
  • 4
  • 28
  • 47

1 Answers1

5

According to the folks at KNP Labs during one of their trainings, best practice is to create necessary data for a scenario to succeed as a part of the Given or Background so you end up with a step that reads "Given I have 7 phone numbers" and the step definition inserts seven phone numbers that can be relied on for the rest of that scenario.

Of course, that's often not feasible if you wish to run tests against a production site, and the strategies I've seen really vary depending on the amount of specific data involved and how volatile the data is on production.

Since best practice also dictates that the feature files should describe the application behavior in terms the feature beneficiary can reasonably be expected to understand, it's unlikely that anything exposing environment-conditional data in the feature file would be an optimal approach. The target feature user probably isn't aware of the varying environments.

If the data on production is stable enough to write tests against, I'd consider setting a parameter or profile in the behat.yml which could be used to indicate the environment at run-time and write a custom step definition. The custom step definition could supply known production values in one case and insert those values in the others. And the Gherkin would still look like "Given I have 7 phone numbers" so that the feature would focus on the business value and the benefit to the user and not the testing environment.

Melissa Anderson
  • 91
  • 2
  • Yeah, this situation is definitely difficult. I do like the idea of finding another way to provide data for the environment by pulling from some custom functions and defining how many to pull in the step call. – Patrick Jul 16 '13 at 15:39
  • 1
    The trick with that though is now we have really started to erode the usefulness of Behat for QA people that are not developers. I am a developer first and a QA person second in this case (being a Software Engineer in Testing). But for the two other QA people on my team, I can't really expect them to know SQL at this point, nor PHP. The idea is to make it as simple as possible to enter the steps and data in and then be able to automate the running of the tests on a GUI-less server in the background. – Patrick Jul 16 '13 at 15:42
  • I will say this, we are starting on a new project that will replace the current system. During this I am pushing heavily for them to give us the methods we need to inject the data into the system on the fly, then test, and finally pull it back out. Unfortunately it will be a year or more before we are entirely moved over to the new system that we are currently building. – Patrick Jul 16 '13 at 15:44
  • As a kind of alongside your idea is another option where instead of inserting the data dynamically (if no mechanisms are available) an alternative would be to query the DB for the data you are looking for and test against whatever you find. Using one of the hook functions to do this. – Patrick Aug 08 '13 at 20:16