Black/gray-box testing of REST API and UI

Question

We are choosing what system for automated acceptance testing to start using (switch) in our company. Currently most of the backend test cases are written by our former tester in Python and for new testers it is difficult to use & maintain it; for UI we use the Robot framework.

I would like to use something standard so that new "typical testers from the street" can start using and yet it should be quite flexible.

In my previous jobs testers used SoapUI or even Apache JMeter with Groovy scripting, but for some reason people in my current company don't like it.

We are considering Fitnesse or Robot framework.

Requirements:

• it should be used for both backend (REST API, some DB checks) and UI testing
• it should use a simple language so even non-programmers/tester can understand the test cases (Product Owners should be able to see whether all acceptance criterias are covered)
• it should support integration with Jenkins
• it should support versioning of test cases so that for a particular product version we also can check out relevant test cases right now we use TestRail (test case management SW); so would be nice if it integrates with it (at least it is possible to program it so send test results there) or completely replaces it

I just quickly played with Fitnesse and to me the tabular form looks pretty ugly. Also at first look the doc is not great (I didn't find possible "commands", e.g. assertions, some looping) and documentation for e.g. the RestFixture is even worse (none).

Also I didn't see any fixture for DB checks. So in the end a developer needs to be involved to program and maintain some custom fixtures, which to me looks worse that use our home grown Python test suite.

Any ideas, experience?

Thanks, Radek

PS: I asked this question also in the QA forum, but it is much less active than StackOverflow, so sorry for this duplication.

Answer 1

I was in an almost similar scenario earlier . we had to decide between RF,fitnesse and IBM'S STAF/STAX

We chose Robot Framework and it worked well.

1. it should be used for both backend (REST API, some DB checks) and UI testing - For REST , RF's requests library along with various DB libraries can be used together .
2. it should use a simple language so even non-programmers/tester can understand the test cases (Product Owners should be able to see whether all acceptance criterias are covered) - RF is intended to do precisely this.
3. it should support integration with Jenkins - RF has a jenkins plugin
4. it should support versioning of test cases so that for a particular product version we also can check out relevant test cases right now - RF's tags feature will work well for this

There exists a robot framework API , so its quite programmable as per the integration requirements.

Answer 2

I can't speak to the use of fitnesse, but robot framework meets all of the things you ask for and more. I choose it for my projects for the following reasons:

1. You can use a single tool (and thus a single reporting format) for SOAP- and REST-based services, database validation, web-based UI testing, and even desktop application testing. It can also be used for integration and unit tests, though there are often better tools for that job.
2. You can use robot tests to implement manual tests using the Dialogs library, or a custom library. I've seen a significant speedup in tester throughput when they ran manual tests written in robot than when running similar tests written in Microsoft Word. Unfortunately there's not much written on the web about this powerful feature, but you can get the same sort of reporting, version control, tagging, etc. features for all of your acceptance tests, both manual and automated.
3. If you invest the time in creating a good keyword library, tests can be easily readable (and writeable!) by non-testers
4. There is a robot plugin for jenkins that makes browsing test results easy
5. Robot framework test suites are plain text files, so they can be versioned right alongside your code.
6. Test output is a very simple to understand and parse XML file. It can also generate xunit-style output for integration with other tools. Robot also comes with tools for converting this xml to human-friendly logs and reports. The listener interface makes it easy to capture or stream live test results.
7. There are a growing number of tools and editor plugins that work with Robot, so your team members can use the tools they are most comfortable with.
8. Robot is very extensible -- keyword libraries can be written in almost any language -- natively in python, as well as java if you run with jython, or .NET languages if you run with IronPython. And with the remote library interface, you can write keywords in any language that can open a socket and act as a server.

As for fixtures for DB tests, there is a generic java-based database library, and a generic python based database library that can connect to just about any common database. There is also a library specifically for talking to MongoDB.

Related to the question you had about versioning, robot has a very powerful tagging mechanism that you might find useful. You could, for example, tag all of the tests with the version of the product that they go with. Then, you just check out everything but use robots command-line options to select only the tests tagged with a particular version. As a side benefit of the tagging, the reports break down the pass/fail statistics by tag.

Robot isn't a perfect test system, but it is a very good one. I would argue that there are many equally good test frameworks, but I'm not sure there are any that are objectively better. Certainly for the things you listed that are important to you, robot framework does everything you need.