4

I am trying to come up with a good tolerance when comparing doubles in unit tests.

If I allow a fixed tolerance as I've seen mentioned on this site (eg return abs(actual-expected) < 0.00001;), this will frequently fail when numbers are very big due to the nature of floating point representation.

If I use a relative tolerance in terms of % error allowed (eg return abs(actual-expected) < abs(actual * 0.001); this fails too often for small numbers (and for very small numbers, the computation itself can introduce rounding error). Additionally, it allows too much tolerance in certain ranges (eg comparing 2000 and 2001 would pass).

I'm wondering if there's any standard algorithm for allowing tolerance that will work for both small and large numbers. Should I try for some kind of base 2 logarithmic tolerance to mirror floating point storage? Should I do a hybrid approach based on the size of the inputs?

Since this is in unit test code, performance is not a big factor.

Jon Brooks
  • 2,472
  • 24
  • 32

1 Answers1

4

The specification of tolerance is a business function. There aren't standards that say "all tolerance must be within +/- .001%". So you have to go to your requirements and figure out what's going to work for you.

Look at your application. Let's say it's for some kind of cutting machine. Are they metal machining tolerances? .005 inches is common. Are they wood cabinet sawing tolerances? 1/32" is sloppy, 1/64" is better. Are they house framing tolerances? Don't expect a carpenter to come closer than 1/4". Hand cutting with a welding torch? Hope for about an inch. The point is simply that every application depends on something different, even when they're doing equivalent things.

If you're just talking "doubles" in general, they're usually good to no better than 15 digits of precision. Floats are good to 7 digits. I round those down by one when I'm thinking about the problem (I don't rely on a double being accurate to more than 14 digits and I stop with floats at six digits); however, if I'm worried about more than the 12th digit of precision I'm generally working with large dollar amounts that have to balance precisely, and I'd be a fool to use non-integer math for them. Business people want their stuff to balance to the penny, and wouldn't approve of rounding off addition operations!

If you're looking at math library operations such as the trig functions, read the library's documentation on each function.

John Deters
  • 4,295
  • 25
  • 41
  • I should have provided some context: I am working on educational math software, so the range of values can really span the entire range of doubles. No matter what I choose, there will be ways to construct false positives. I was really looking for a generic algorithm for saying 'these two doubles are pretty close to each other' that will be equally good for the entire range of doubles. I am fascinated by the integer-representation strategy here: http://www.cygnus-software.com/papers/comparingfloats/comparingfloats.htm but don't think it's viable in Javascript, which this project is in... – Jon Brooks Oct 09 '13 at 23:09
  • @JonBrooks, Regarding the "size of fail", you're overestimating the importance of your tolerance factor. It doesn't matter if it's "perfect", because it's already near the boundary of imperfection. Bits and differences cause jitter, and anybody who steps near that jittery line has already strayed considerably from the accurate answer. What matters is that it gets the job done appropriately (efficient, readable, usable, etc.) – John Deters Oct 10 '13 at 19:21
  • @JonBrooks, Personally, I'd use the % algorithm and aim for 0.0001 % accuracy throughout. However, in a math class you will have different times when different levels of approximation are close enough (the business cases are different.) Your algorithm really should be adjustable, tunable to the specific problems the users are working on. – John Deters Oct 10 '13 at 19:22
  • This turns out to be pretty good advice. I decided that there wasn't a one-shot answer: I went with relative precision for calculations, but fixed precision for placement of geometric objects on the screen - where we care about being as close to (0,0) as we do about (1000, 1000). Thanks again for your answer. – Jon Brooks Oct 11 '13 at 20:15