1

What are the different metrics we use for assuring the quality of test suites written based on only requirements and specifications (black box)?

Simply put, given a set of requirements and a test suite on those requirements, what are different metrics to quantify the quality of specification/requirement based testing (test suite)?

I read through the following articles regarding specification based testing and metrics to define them. These topics are too abstract to digest.

http://link.springer.com/chapter/10.1007%2F978-3-642-21768-5_13#page-1 http://www.worldscientific.com/doi/abs/10.1142/S0218539301000530

Can you please explain in simple terms?

Thanks!

Dave Schweisguth
  • 36,475
  • 10
  • 98
  • 121
chebus
  • 762
  • 1
  • 8
  • 26

1 Answers1

0

The simplest way to evaluate specification-based testing is to trace each specification to a test (whether manual or automated), count which specifications are tested and which are not, and calculate percent coverage.

The confusion related to the articles you linked to is due to confusion between "specification" used to refer to a human-written, structured but relatively informal document, and "specification" meaning a formal computer-readable specification from which tests can be automatically derived.

It's also possible to measure code coverage during specification-based testing. However, it's very difficult to improve coverage without looking inside the black box. Also, specification-based tests are slow, even when automated, so it's painful to achieve code coverage using only specification-based tests. A better approach is to combine black-box specification-based tests and white-box unit tests and consider overall coverage.

Dave Schweisguth
  • 36,475
  • 10
  • 98
  • 121