How to test multi-parameter formula

Question

I'm refactoring some code that implements a formula and I want to do it test-first, to improve my testing skills, and leave the code covered.

This particular piece of code is a formula that takes 3 parameters and returns a value. I even have some data tables with expected results for different inputs, so in theory, I could jusst type a zillion tests, just changing the input parameters and checking the results against the corresponding expected value.

But I thought there should be a better way to do it, and looking at the docs I've found Value Parameterized Tests.

So, with that I now know how to automatically create the tests for the different inputs.
But how do I get the corresponding expected result to compare it with my calculated one?

The only thing I've been able to come up with is a static lookup table and a static member in the text fixture which is an index to the lookup table and is incremented in each run. Something like this:

#include "gtest/gtest.h"

double MyFormula(double A, double B, double C)
{
    return A*B - C*C;   // Example. The real one is much more complex
}

class MyTest:public ::testing::TestWithParam<std::tr1::tuple<double, double, double>>
{
protected:

    MyTest(){ Index++; }
    virtual void SetUp()
    {
        m_C = std::tr1::get<0>(GetParam());
        m_A = std::tr1::get<1>(GetParam());
        m_B = std::tr1::get<2>(GetParam());
    }

    double m_A;
    double m_B;
    double m_C;

    static double ExpectedRes[];
    static int Index;

};

int MyTest::Index = -1;

double MyTest::ExpectedRes[] =
{
//               C = 1
//      B:   1     2     3     4     5     6     7     8     9    10
/*A =  1*/  0.0,  1.0,  2.0,  3.0,  4.0,  5.0,  6.0,  7.0,  8.0,  9.0, 
/*A =  2*/  1.0,  3.0,  5.0,  7.0,  9.0, 11.0, 13.0, 15.0, 17.0, 19.0, 
/*A =  3*/  2.0,  5.0,  8.0, 11.0, 14.0, 17.0, 20.0, 23.0, 26.0, 29.0, 

//               C = 2
//      B:     1     2     3     4     5     6     7     8     9    10
/*A =  1*/   -3.0, -2.0, -1.0,  0.0,  1.0,  2.0,  3.0,  4.0,  5.0,  6.0, 
/*A =  2*/   -2.0,  0.0,  2.0,  4.0,  6.0,  8.0, 10.0, 12.0, 14.0, 16.0, 
/*A =  3*/   -1.0,  2.0,  5.0,  8.0, 11.0, 14.0, 17.0, 20.0, 23.0, 26.0, 
};

TEST_P(MyTest, TestFormula)
{
    double res = MyFormula(m_A, m_B, m_C);
    ASSERT_EQ(ExpectedRes[Index], res);
}

INSTANTIATE_TEST_CASE_P(TestWithParameters,  
                        MyTest,  
                        testing::Combine( testing::Range(1.0, 3.0), // C
                                          testing::Range(1.0, 4.0), // A 
                                          testing::Range(1.0, 11.0) // B
                                          ));

Is this a good approach or is there any better way to get the right expected result for each run?

score 14 · Answer 1 · answered Jan 24 '12 at 23:24

Include the expected result along with the inputs. Instead of a triple of input values, make your test parameter be a 4-tuple.

class MyTest: public ::testing::TestWithParam<
  std::tr1::tuple<double, double, double, double>>
{ };

TEST_P(MyTest, TestFormula)
{
  double const C = std::tr1::get<0>(GetParam());
  double const A = std::tr1::get<1>(GetParam());
  double const B = std::tr1::get<2>(GetParam());
  double const result = std::tr1::get<3>(GetParam());

  ASSERT_EQ(result, MyFormula(A, B, C));
}

The downside is that you won't be able to keep your test parameters concise with testing::Combine. Instead, you can use testing::Values to define each distinct 4-tuple you wish to test. You might hit the argument-count limit for Values, so you can split your instantiations, such as by putting all the C = 1 cases in one and all the C = 2 cases in another.

INSTANTIATE_TEST_CASE_P(
  TestWithParametersC1, MyTest, testing::Values(
    //           C     A     B
    make_tuple( 1.0,  1.0,  1.0,  0.0),
    make_tuple( 1.0,  1.0,  2.0,  1.0),
    make_tuple( 1.0,  1.0,  3.0,  2.0),
    // ...
  ));  

INSTANTIATE_TEST_CASE_P(
  TestWithParametersC2, MyTest, testing::Values(
    //           C     A     B
    make_tuple( 2.0,  1.0,  1.0, -3.0),
    make_tuple( 2.0,  1.0,  2.0, -2.0),
    make_tuple( 2.0,  1.0,  3.0, -1.0),
    // ...
  ));

Or you can put all the values in an array separate from your instantiation and then use testing::ValuesIn:

std::tr1::tuple<double, double, double, double> const FormulaTable[] = {
  //           C     A     B
  make_tuple( 1.0,  1.0,  1.0,  0.0),
  make_tuple( 1.0,  1.0,  2.0,  1.0),
  make_tuple( 1.0,  1.0,  3.0,  2.0),
  // ...
  make_tuple( 2.0,  1.0,  1.0, -3.0),
  make_tuple( 2.0,  1.0,  2.0, -2.0),
  make_tuple( 2.0,  1.0,  3.0, -1.0),
  // ...
};

INSTANTIATE_TEST_CASE_P(
  TestWithParameters, MyTest, ::testing::ValuesIn(FormulaTable));

score 1 · Answer 2 · answered Jan 23 '12 at 17:46

1

See hard coding the expected result is like you are limiting again the no of test cases. If you want to get a complete data driven model, I would rather suggest you to read inputs, expected result from a flat file/xml/xls file.

answered Jan 23 '12 at 17:46

Pritam Karmakar

2,773
5
30
49

Thanks. In this case, the specification I have to follow provides the formula plus some tables with expected results for some values. I can be quite confident that if my results match those tables, I've done it ok. In fact, I've already found some wrong values in the specification, thanks to the tests. – MikMik Jan 24 '12 at 06:37
That's great. If you add your test data within your test class, the problem will be, in future if you want to test another scenario then you have to update your class again. But if you access your test data from an external file then your test case maintainability will be reduced. Because you are going to update that flat file to add a new test data. that's it. No need to build your class and then again deploy it. – Pritam Karmakar Jan 24 '12 at 08:00
@PritamKarmakar, you mean that the test case maintainability will be increased, right? The effort required to maintain it will be reduced. – Alan May 25 '17 at 14:09

score 0 · Answer 3 · answered Jan 23 '12 at 17:36

I don't have much experience with unit testing, but as a mathematician, I think there is not a lot more you could do.

If you would know some invariants of your formula, you could test for them, but i think that does only make sense in very few scenarios.

As an example, if you would want to test, if you have correctly implemented the natural exponential function, you could make use of the knowledge, that it's derivative should have the same value as the function itself. You could then calculate a numerical approximation to the derivative for a million points and see if they are close to the actual function value.

How to test multi-parameter formula

3 Answers3