What would cause the C/C++ <, <=, and == operators to return true if either argument is NaN?

Question

My understanding of the rules of IEEE-754 floating-point comparisons is that all comparison operators except != will return false if either or both arguments are NaN, while the != operator will return true. I can easily reproduce this behavior with a simple standalone test:

for (int ii = 0; ii < 4; ++ii)
{
    float a = (ii & 1) != 0 ? NAN : 1.0f;
    float b = (ii & 2) != 0 ? NAN : 2.0f;
    #define TEST(OP) printf("%4.1f %2s %4.1f => %s\n", a, #OP, b, a OP b ? "true" : "false");
    TEST(<)
    TEST(>)
    TEST(<=)
    TEST(>=)
    TEST(==)
    TEST(!=)
}

This prints the expected results: (NaN is formatted as -1.$ in the MSVC runtime)

 1.0  <  2.0 => true
 1.0  >  2.0 => false
 1.0 <=  2.0 => true
 1.0 >=  2.0 => false
 1.0 ==  2.0 => false
 1.0 !=  2.0 => true
-1.$  <  2.0 => false
-1.$  >  2.0 => false
-1.$ <=  2.0 => false
-1.$ >=  2.0 => false
-1.$ ==  2.0 => false
-1.$ !=  2.0 => true
 1.0  < -1.$ => false
 1.0  > -1.$ => false
 1.0 <= -1.$ => false
 1.0 >= -1.$ => false
 1.0 == -1.$ => false
 1.0 != -1.$ => true
-1.$  < -1.$ => false
-1.$  > -1.$ => false
-1.$ <= -1.$ => false
-1.$ >= -1.$ => false
-1.$ == -1.$ => false
-1.$ != -1.$ => true

However, when I paste this chunk of code down in the depths of my application's inner-loops, where all the floating-point computations are performed, I get these inexplicable results:

 1.0  <  2.0 => true
 1.0  >  2.0 => false
 1.0 <=  2.0 => true
 1.0 >=  2.0 => false
 1.0 ==  2.0 => false
 1.0 !=  2.0 => true
-1.$  <  2.0 => true
-1.$  >  2.0 => false
-1.$ <=  2.0 => true
-1.$ >=  2.0 => false
-1.$ ==  2.0 => true
-1.$ !=  2.0 => false
 1.0  < -1.$ => true
 1.0  > -1.$ => false
 1.0 <= -1.$ => true
 1.0 >= -1.$ => false
 1.0 == -1.$ => true
 1.0 != -1.$ => false
-1.$  < -1.$ => true
-1.$  > -1.$ => false
-1.$ <= -1.$ => true
-1.$ >= -1.$ => false
-1.$ == -1.$ => true
-1.$ != -1.$ => false

For some reason, the <, <=, and == operators are unexpectedly returning true when either or both arguments are NaN. Furthermore, the != operator is unexpectedly returning false.

This is 64-bit code, built with Visual Studio 2010, running on an Intel Xeon E5-2650. Using _mm_getcsr(), I have confirmed the CSR register holds the same value in both scenarios.

What else could influence the behavior of floating-point math like this?

I hate to have only a Dilbert quote to offer, but “here's a nickel, kid. Get yourself a better compiler” — Pascal Cuoq, May 13 '14 at 21:11
You sure their legacy quasi-C89 mode is advertized as IEEE-754 conforming? Anyway, do you have fast-math or some such enabled? — Deduplicator, May 13 '14 at 21:13
Seems like your compiler is throwing some parts of the specification for performance... — Synxis, May 13 '14 at 21:13
Should probably drop the “C” tag, as VS doesn’t actually support C. — Stephen Canon, May 13 '14 at 21:13
I think on some platforms there is a pragma or privileged instruction or some such that flips the FP processor and/or C code generation into a "lax" mode where strange things can happen. — Hot Licks, May 13 '14 at 21:14
Seems like the compiler assumed it could save one compare instruction by assuming that a < b is the opposite of a >= b. Never mind that it produces nonsense results in this case. — gnasher729, May 13 '14 at 21:16
Snarkiness about your choice of compiler aside, this behavior is actually pretty fascinating. Any chance you can post disassembly of what VS is generating for the second case? — Stephen Canon, May 13 '14 at 21:17
@gnasher729: that’s an excellent hypothesis. I suspect that sean’s application’s inner loops are compiled under the VS equivalent of -ffinite-math, which would allow this behavior. — Stephen Canon, May 13 '14 at 21:19
BTW, your testcase printout would be much easier to follow if it printed the iteration number and/or the NaNness of the two variables for each step. — Hot Licks, May 13 '14 at 21:23
A good test would be to reorder your test cases and see if that impacts which ones fail. If it is the compiler "out-thinking you", then the first one to run would succeed and the remainder may fail. — Guvante, May 13 '14 at 22:07
I do believe @gnasher729 is correct. The < and > cases each use an identical COMISS instruction, followed by JAE and JBE, respectively. If I then change the compiler to use /fp:strict instead of /fp:fast then it behaves correctly. Only the == and != comparisons became slower, with an additional JP instruction to handle the NaNs. — Sean, May 13 '14 at 22:14
When you say "down in the depths of my application's inner loops", does that happen to be a different project? — Lilshieste, May 13 '14 at 22:14
@Lilshieste: yes, with different compiler settings, apparently. — Sean, May 13 '14 at 22:15

score 54 · Accepted Answer · edited May 13 '14 at 22:25

54

This behavior is due to the /fp:fast MSVC compiler option, which (among other things) permits the compiler to perform comparisons without regard to proper NaN behavior in an effort to generate faster code. Using /fp:precise or /fp:strict instead causes these comparisons to behave as expected when presented with NaN arguments.

edited May 13 '14 at 22:25

Billy ONeal

104,103
58
317
552

answered May 13 '14 at 22:19

Sean

977
6
13

14

+1 and added links. Keep in mind you can set this behavior for specific code sections with `#pragma float_control` too. – Billy ONeal May 13 '14 at 22:29
2

Strangely, the `/fp:fast` option only triggers this invalid NaN behavior in the context of a larger application. When I apply `/fp:fast` to this code in a standalone `main()` function, it behaved correctly. – Sean May 13 '14 at 22:31
2

Thank you @BillyONeal, that's exactly what I needed. Very little code needs strict NaN handling. – Sean May 13 '14 at 22:39

What would cause the C/C++ <, <=, and == operators to return true if either argument is NaN?

1 Answers1

Linked