Large lua numbers are being printed incorrectly

Question

I have the following test case:

Lua 5.3.2  Copyright (C) 1994-2015 Lua.org, PUC-Rio
> foo = 1000000000000000000
> bar = foo + 1
> bar
1000000000000000001
> string.format("%.0f", foo)
1000000000000000000
> string.format("%.0f", bar)
1000000000000000000

That last line should be 1000000000000000001, since that's the value of bar, but for some reason it's not. This doesn't only apply to 1000000000000000000, I've yet to find another number over that one which gives the correct value. Can anyone give an explanation for why this happens?

score 5 · Answer 1 · answered Apr 17 '16 at 18:25

You're formatting the number as floating-point, not integer. That's what %.0f is doing. At some point, floats lose precision. double, for example, will lose precision after about 16 decimal digits.

If you want to format an integer as an integer, then you need to format it as an integer, using standard printf rules:

string.format("%i", bar)

score 3 · Accepted Answer · answered Apr 17 '16 at 18:32

log2(1000000000000000000) is between 59 and 60, which means that the binary representation of that number needs 60 bits. double-precision floating point numbers have only 53 bits of precision, plus a power-of-two exponent with 11 bits of range. So to store that large of a number as floating point (which is what you requested with the %f format specifier), six to seven bits of precision are chopped off the end of the number, and the whole thing is multiplied by a power of two to get it back in range (2⁵⁹ in this case, I think). Chopping off those final bits removes the precision that allows 1000000000000000000 and 1000000000000000001 to be distinct from each other.

(This is not a particularly precise description of floating point, apologies if my numbers or descriptions are not exact.)

Large lua numbers are being printed incorrectly

2 Answers2