sprintf : fixed point, big numbers and precision loss

Question

I need to read some numbers in a database and write them into a text file using Perl.

In the table where are the numbers, the data format is defined as numeric (25,5) (it reads 25 digits, including 5 decimals).

I format the numbers in my file with a sprintf "%.5f", $myvalue to force 5 decimals and I just noticed that for greats values, there is a precision loss for numbers with more than 17 digits :

db   = 123.12345
file = 123.12345 (OK)

db   = 12345678901234891.12345
file = 12345678901234892.00000 (seems to be rounded to upper integer)

db   = 12345678901234567890.12345
file = 12345678901234567000.00000 (truncation ?)

What is Perl's greatest precision for fixed decimal numbers?

I am aware of the concepts and limitations of floating point arithmetic in general, but I am not a Perl monk and I do not know the internals of Perl so I don't know if it is normal (or if it is related at all to floating point). I am not sure either if it is a internal limitation of Perl, or a problem related to the sprintf processing.

Is there a workaround or a dedicated module that could help with that problem?

Some notable points :

this is an additional feature of a system that already uses Perl, so using another tool is not an option
the data being crunched is financial so I need to keep every cent and I cannot cope with a +/- 10 000 units precision :^S

Is this premature optimization? Why do you feel the need to "force" 5 decimals? — TLP, Jul 25 '13 at 11:44
When you write strings to a text file, you are not optimizing. — 7stud, Jul 25 '13 at 11:53
@choroba I was thinking [`Math::BigFloat`](http://perldoc.perl.org/Math/BigFloat.html), it seems to be able to at least print the number correctly. — TLP, Jul 25 '13 at 11:53
@TLP: it just the output that is required. Unsure if there is an actual and objective reason, the file will be automatically processed later, not directly used by an human. And `Math::BigFloat` did helped, I found that right after asking my question. — Seki, Jul 25 '13 at 12:02
@Seki And these automatic processes can't handle numbers with less than 5 decimals? I think I would focus on solving the error from that end instead of playing around with sprintf and risk losing precision. — TLP, Jul 25 '13 at 12:05
@7stud: could you develop? You mean that the 'stringified' value could been written into the file by another mean than getting the string representation of the value? — Seki, Jul 25 '13 at 12:06
Yes, I found that `Math::BigFloat` seemed to help too, but not knowing why it seemed to work, it seemed like a dangerous thing to suggest. I did notice, however, that this truncation happens even if you do `perl -nle 'print 0+$_'` with the input being that big float. — TLP, Jul 25 '13 at 12:08
@Seki What I am wondering, though, is if you have the hang of Perl's automatic conversion. When you print the number that comes directly from the DB, do you then notice it is not printing the 5 decimals? I would assume the precision is identical. — TLP, Jul 25 '13 at 12:11
@TLP: unfortunately, I do not have my word to change that point. The other end of the line is an administration and it is not the first time that we need to follow to the letter the specifications, even to provide some nonsense data format just because their processor is rather narrow-minded and hard-coded... — Seki, Jul 25 '13 at 12:14

score 2 · Accepted Answer · answered Jul 25 '13 at 11:57

2

Once again, I am finding a solution right after asking SO. I am putting my solution here, to help a future visitor :

replace

$myout = sprintf "%.5f", $myvalue;

by

use Math::BigFloat;
$myout = Math::BigFloat->new($myvalue)->ffround( -5 )->bstr;

answered Jul 25 '13 at 11:57

Seki

11,135
7
46
70

score 2 · Answer 2 · answered Jul 25 '13 at 12:11

Without modules like Math::BigFloat, everything above 16 digits is pure magic... e.g.

perl -e 'printf "*10^%02d: %-.50g\n", $_, log(42)*(10**$_) for (0..20)'

produces

*10^00: 3.7376696182833684112267746968427672982215881347656
*10^01: 37.376696182833683224089327268302440643310546875
*10^02: 373.76696182833683224089327268302440643310546875
*10^03: 3737.6696182833684360957704484462738037109375
*10^04: 37376.6961828336861799471080303192138671875
*10^05: 373766.96182833681814372539520263671875
*10^06: 3737669.6182833681814372539520263671875
*10^07: 37376696.18283368647098541259765625
*10^08: 373766961.82833683490753173828125
*10^09: 3737669618.283368587493896484375
*10^10: 37376696182.83368682861328125
*10^11: 373766961828.33685302734375
*10^12: 3737669618283.36865234375
*10^13: 37376696182833.6875
*10^14: 373766961828336.8125
*10^15: 3737669618283368.5
*10^16: 37376696182833688
*10^17: 373766961828336832
*10^18: 3737669618283368448
*10^19: 37376696182833684480
*10^20: 373766961828336828416

score 1 · Answer 3 · answered Jul 25 '13 at 11:54

1

What is Perl's greatest precision for fixed decimal numbers?

Perl doesn't have fixed point decimal numbers. Very few languages do, actually. You could use a module like Math::FixedPoint, though

answered Jul 25 '13 at 11:54

ikegami

367,544
15
269
518

So the internal representation for a decimal value in Perl is actually a float? – Seki Jul 25 '13 at 12:16
2

@Seki, yup. (Well, actually a double, possibly a quad) – ikegami Jul 25 '13 at 13:38
@ikegami 12345678901234892 is the closest IEEE 754 64-bit binary value to 12345678901234891.12345, suggesting double. – Patricia Shanahan Jul 25 '13 at 14:11
@Patricia Shanahan, It's not the same for all builds. Per uses doubles and possibly quads. – ikegami Jul 25 '13 at 16:25

score 1 · Answer 4 · answered Jul 25 '13 at 16:19

Perl is storing your values as floating-point numbers internally.¹ The precision is dependent on how your version of Perl is compiled, but it's probably a 64-bit double.

C:\>perl -MConfig -E "say $Config::Config{doublesize}"
8

A 64-bit double-precision float² has a 53-bit significand (a.k.a. fraction or mantissa) which gives it approximately 16 decimal characters of precision. Your database is defined as storing 25 characters of precision. You'll be fine if you treat the data as a string but if you treat it as a number you'll lose precision.

Perl's bignum pragma provides transparent support for arbitrarily large numbers. It can slow things down considerably so limit its use to the smallest possible scope. If you want big floats only (without making other numeric types "big") use Math::BigFloat instead.

_{1. Internally, perl uses a datatype called an SV that can hold floats, ints, and/or strings simultaneously.

2. Assuming IEEE 754 format.}

score 0 · Answer 5 · answered Mar 03 '16 at 20:12

Alternatively, if you're just transferring the values from the database to a text file and not operating on them as numbers, then have the DB format them as strings. Then read and print them as strings (perhaps using "printf '%s'"). For example:

select Big_fixed_point_col(format '-Z(24)9.9(5)')(CHAR(32))

sprintf : fixed point, big numbers and precision loss

5 Answers5