3

So I've read a lot on this, so please know that I get that a number like 0.6 can not be represented absolutely accurately as a Java double - but I do know that there is a version of a double that represents the number 0.6 closely enough that the number gets displayed as 0.6 instead of something like 0.6000000000000001 in most end interpretations of that number (e.g. including toString() or marshaling to JSON using a library like Jackson - my goal).

I'm looking for a mechanism to coerce a double into a number that is interpreted as a relative truncated precision. E.g., if I want to truncate to 0.1 precision, I want something that will represent 0.6975265613 truncated to a 0.1 precision as 0.6, not 0.6000000000000001

So, basically making the following test work:

@Test
public void testRounding() {

    // prove there is a value that displays cleanly
    Double cleanValue = 0.6d;
    assertEquals("0.6", cleanValue.toString());

    Double value = 0.6975265613d;
    Double precision = 0.1d;

    value = value / precision;
    value = value.longValue() * precision;

    // this fails with 0.6000000000000001
    assertEquals("0.6", value.toString());

}

I'm using the longValue() to truncate for speed, but you get the same result with Math.floor().

Looking at the raw double hex values, the cleanValue above is: 3fe3333333333333

While the result of the rounding (value) is: 3fe3333333333334

I need a technique that will consistently give me the intended version (the first), regardless of the rounding precision - assuming the precision doesn't push the limit of precision for the type. Conversion to String or BigDecimal will be too slow - this is an analytic app where the x10 cost of these methods will be very measurable.

Again, understood that neither hex number really represents the actual number 0.6, I just want Java to think it's 0.6 (i.e. 3fe3333333333333) :)

Thanks!

tsquared
  • 119
  • 6
  • So what you're trying to do is to find integers `x` and `y` such that `x / 10^y` is as close to the double as possible. So it's a two-dimensional optimization problem. – Mysticial May 29 '15 at 02:44
  • 1
    Any reason you don't want to use BigDecimal? – BevynQ May 29 '15 at 02:44
  • 1
    I'd recommend not storing the precision as `0.1d` - you're eating a roundoff error there that would be easy to avoid. – user2357112 May 29 '15 at 02:50
  • I don't think there's anything practical about this. There are too many ways a double can get converted to decimal for a single technique to be reliable at the double end. What you should be focussing on is the conversion process itself, which is easy to control, and making sure you always use it. – user207421 May 29 '15 at 02:56
  • I dunno, seems practical enough to me. We demand that operations like `+` and `/` are correctly rounded; why not demand the same of rounding? It's not like `exp`, where there's a compelling reason to settle for accuracy within one ULP. – user2357112 May 29 '15 at 03:01
  • @user2357113 The rules of floating-point are already defined. 'We' aren't at liberty to 'demand ' anything. And the 'correct' rounding of + and / already disagrees with the objective here. – user207421 May 29 '15 at 03:06
  • @EJP: "Correctly rounded" means returning a result always accurate within 0.5 ULPs, not the result produced by the algorithm in the OP, even if the steps of that algorithm are correctly rounded individually. The correct rounding of + and / does not disagree with the objective; the way the operations have been applied is in conflict with the objective. – user2357112 May 29 '15 at 03:19
  • @tsquared: You say that converting to String will be too slow, but your goal is marshalling to JSON. You'll need to do a string conversion anyway, and probably file or network IO. Are you sure the cost of converting to String or BigDecimal will be too high? This is a tricky thing to get right otherwise. – user2357112 May 29 '15 at 03:24
  • Also, `0.6d` is mathematically slightly smaller than the real number 0.6. Do you want to truncate this to `0.6d`, or `0.5d`? Whichever way you choose, I suspect it'll be crucial that it never goes the other way. – user2357112 May 29 '15 at 05:12
  • Comment on speed - the rounding happens across 10s of millions of entities - the result is marshaled, see below for speed impact of BigDecimal conversion. Thank you to @user2357112 for being the only one to avoid the religious debate on how I should just accept the preordained limitations of floating point, embrace it, and just put 0.600000000000000001 on my screen for my users because that's the way the founding fathers of binary computers wanted it. Still searching for a practical solution. – tsquared May 29 '15 at 12:06

1 Answers1

1

This passes not sure it is what you want though

@Test
public void testRounding() {

    // prove there is a value that displays cleanly
    Double cleanValue = 0.6d;
    assertEquals("0.6", cleanValue.toString());

    Double value = 0.6975265613d;

    // get double value truncated to 1 decimal place
    value = BigDecimal.valueOf(value).setScale(1,BigDecimal.ROUND_DOWN).doubleValue();

    // this fails with 0.6000000000000001
    assertEquals("0.6", value.toString());

}
BevynQ
  • 8,089
  • 4
  • 25
  • 37
  • Thanks for the response, but as I said, BigDecimal is too slow. E.g.: running my original conversion takes 17ms for 100M iterations while the BigDecimal approach takes 47,333ms on my machine... so more like a 3000x penalty than a 100x penalty. Just won't work. – tsquared May 29 '15 at 11:55
  • @tsquared: How long does it take to marshal the data and write it to disk or send it across the network or whatever? – user2357112 May 29 '15 at 19:05
  • @tsquared: Also, have you considered providing Jackson with your own custom serializer for doubles? – user2357112 May 29 '15 at 19:14
  • I realize I have the slightly unique need to run through millions of records and bin the data points - so I need to round at high speed regardless of the marshaling of the result. It would be more elegant to round to the value Java considers 0.6 instead of one bit off from that, but after a lot of messing around with algorithms to try to predict what that would be for all values (try reading Sun's FloatingPointDecimal.dtoa() function) - I realized it's pretty much indecipherable. If I can get the binning to consistently go to 0.6000..001 then I can clean it up on marshaling. Thanks! – tsquared May 30 '15 at 12:47