comparing floating point numbers in C for unit testing

Question

So I'm using CUnit to do unit testing. I expect something like

float x;
x = atof("17.99");

And I want to test this with an assert; obviously with some small epsilon I could

CU_ASSERT(abs(x - atof("17.99")) < epsilon);

However I'm considering

r = atof("17.99");
CU_ASSERT(0 == memcmp(&x, &r, sizeof(float));

This does appear to work. I wish to use this to avoid having to manually set epsilon on each test based on the values. In the above case 1e-6 should be sufficient; however if the value is 1e-10 using epsilon of 1e-6 may not catch a problem. The more choices the developer has to make the more room for error.

My questions are: Should this technique be stable on posix systems? That is, if the two floating point numbers being compared are generated by exactly the same steps should their internal representation be exactly the same.

edit: More to the point I'd eventually like a CU_ASSERT_FLOAT_EQUAL macro.

Yes, if they are generated by the same steps, their representation will be the same. Unless the result is a NaN, you'll have `x == y` then. — Daniel Fischer, May 29 '12 at 13:42
What problem does `memcmp()` solve that a simple `==` wouldn't? — mouviciel, May 29 '12 at 13:45
@DanielFischer: Is that guaranteed? I'd assume it is, but what about potential compiler optimizations (intermediate results in wide floating-point registers, etc.)? — Oliver Charlesworth, May 29 '12 at 13:46
@mouviciel: for floating point types using == does not work, even with identical representations. See http://dl.acm.org/citation.cfm?id=103163 — Michael Conlen, May 29 '12 at 13:51
@DanielFischer: that's a good point; one might need to assume that the code generating the test and the code generating the object file for the tested function are compiled using the same optimizations. — Michael Conlen, May 29 '12 at 13:51
@MichaelConlen With equal repesentation, you definitely get `==` to be true, unless you fool around with `NaN`. — glglgl, May 29 '12 at 14:00
@glglgl: this was not the case on Darwin Euler.local 11.3.0 Darwin Kernel Version 11.3.0: Thu Jan 12 18:47:41 PST 2012; root:xnu-1699.24.23~1/RELEASE_X86_64 x86_64 with gcc version 4.2.1 (Based on Apple Inc. build 5658) (LLVM build 2336.1.00) — Michael Conlen, May 29 '12 at 14:03
In my particular case == returned false, subtracting one from the other returned -0.00000 and fabs(x-atof("17.99")) < 1e-6 was true — Michael Conlen, May 29 '12 at 14:04
Not quite a dupe but this might be useful http://stackoverflow.com/questions/3515293/convenient-method-in-googletest-for-a-double-comparison-of-not-equal — Martin Beckett, May 30 '12 at 16:34

Ian Goldby · Answer 1 · 2012-05-31T07:28:26.903

Comparing floating point values is hard. Mixing in strings isn't going to make things any better, and certainly doesn't introduce the small amount of leeway that epsilon would.

Have a look at this article: Comparing Floating Point Numbers, 2012 Edition. For my money, ULP is the way to go. There are some nasty edge cases, but you can probably ignore most of them.

Edit: 31 May Having thought about this a bit more, I think what you are asking is "If exactly the same steps are used to calculate a float in the test function and in the function under test, will the answer be exactly the same - so that I don't need to worry about +/- some small error?"

The answer is yes, they will be the same. But in that case you have rendered the unit test pointless because if there is a mistake in the code under test then the same mistake will necessarily be present in the test function.

By the way, don't get distracted by the use of memcmp(&x, &y, sizeof(x)). This just tests that exactly the same bits are set in both values. If that is true, then x == y will necessarily be true. Stick with x == y.

score 2 · Answer 2 · answered Jul 01 '12 at 17:20

Beware of comparing a float to a double, as is done in the initial question. The float will necessarily have lost precision, so when it is compared to a full-precision double you may find that the numbers are not equal, even when the calculations are perfect.

See http://randomascii.wordpress.com/2012/06/26/doubles-are-not-floats-so-dont-compare-them/ for details.

score 1 · Answer 3 · answered May 29 '12 at 14:09

float r, x;

r = atof("17.99");
CU_ASSERT(0 == memcmp(&x, &r, sizeof(float));

should have the same effect as

r = atof("17.99");
CU_ASSERT(x == r);

Note that atof returns a double, so

CU_ASSERT(x == atof("17.99"));

is different, but

CU_ASSERT(x == (float)atof("17.99"));

should also be the same.

Note also that gcc optimizer had a long standing bug with this on x86 when using the instructions inherited from the x87 (if I'm not mistaken, that doesn't happen on x86_64).

score 1 · Accepted Answer · edited Jan 09 '18 at 14:40

So the answer appears to be no.

The reason is that even if you use an identical series of computations in computing each variable, if there is any code between the steps of computing the value you may induce different rounding errors, as noted by the note in @AProgrammer's response.

The issue is that while you might declare a n-bit floating point it may be stored in a larger register (The x87 uses an 80 bit register). If the value is pushed off the register and into memory to free the register for other operations, the value is then truncated (rounded? Where did my notes go...). When the value is brought back on to the register that lost precision carries through the rest of the computation.

On the other hand another piece of code may go through the exact same steps in computing the value; however if the value is not pushed off the register (or is pushed off at a different place...) then you get a different truncation when it is stored in memory again.

All this is IEEE approved according to the notes from the gcc mailing lists/bug reports.

Since I haven't touched a register since the 80386, I can only guess at what the modern x86 and amd_64 processors have; but I'm guessing without hints to gcc that for the x86 it's using a basic x87 register set or a basic SSE register set.

So the rule of thumb to use fabs(x-y) < epsilon; holds, which for CUnit is provided for in double format (and one could easily write a float version of the macro if one wanted to be as anal about things as I have a habit of getting) as noted by the post which @Martin Beckett commented on.

> generated by exactly the same steps That ends up being extremely tricky to enforce and there are more issues than the ones that you list. If the two runs are on different architectures (even x86 versus x64) or different compilers or different build settings or... For a full discussion of the challenges see this post I wrote: https://randomascii.wordpress.com/2013/07/16/floating-point-determinism/ — Bruce Dawson, Nov 22 '15 at 08:21

score 0 · Answer 5 · answered Jan 22 '20 at 10:20

I think there is a different way of approaching this issue. The question is about unit testing. Unit testing should both test and document the unit under test (uut).

To this end when writing the test one should ask what are the acceptable tolerances for the uut. (This may need to be considered for each uut rather than across the test project).

This way the test can avoid testing for equality, test that the value is within the acceptable range as well as documenting the acceptable tolerance of the unit tests result.

comparing floating point numbers in C for unit testing

5 Answers5