Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
5
votes
3 answers
Why does multiplying and dividing by N "fix" floating point representation?
I am working in JavaScript, but the problem is generic. Take this rounding error:
>> 0.1 * 0.2
0.020000000000000004
This StackOverflow answer provides a nice explanation. Essentially, certain decimal numbers cannot be represented as precisely in…
jds
- 7,910
- 11
- 63
- 101
5
votes
2 answers
simple floating-point numbers lose precision
I'm using Delphi XE2 Update 3. There are precision issue with even the simplest of floating-point numbers (like 3.7). Given this code (a 32-bit console app):
program Project1;
{$APPTYPE CONSOLE}
{$R *.res}
uses System.SysUtils;
var s: Single;…
James L.
- 9,384
- 5
- 38
- 77
5
votes
3 answers
Print __float128, without using quadmath_snprintf
In my question about Analysis of float/double precision in 32 decimal digits, one answer said to take a look at __float128.
I used it and the compiler could find it, but I can not print it, since the complier can not find the header quadmath.h.
So…
gsamaras
- 71,951
- 46
- 188
- 305
5
votes
2 answers
Javascript: string representation of numbers
How does javascript convert numbers to strings? I expect it to round the number to some precision but it doesn't look like this is the case. I did the following tests:
> 0.1 + 0.2
0.30000000000000004
> (0.1 +…
martinkunev
- 1,364
- 18
- 39
5
votes
4 answers
Formatting Floating point number in java upto 3 precison of decimal
I have a variable type float.I wanted to print it upto 3 precision of decimal including trailing zeros.
Example :
2.5 >> 2.500
1.2 >> 1.200
1.3782 >> 1.378
2 >> 2.000
I am trying it by using
DecimalFormat _numberFormat= new…
Abhay
- 687
- 4
- 13
- 22
5
votes
3 answers
Is it 52 or 53 bits of floating point precision?
I keep on seeing this nonsense about 53 bits of precision in 64-bit IEEE floating point representation. Would someone please explain to me how in the world a bit that is stuck with a 1 in it contributes ANYTHING to the numeric precision? If you…
Chris Cochran
- 321
- 2
- 9
5
votes
4 answers
Is std::greater and std::less safe to use?
When comparing double values in C++, using the <,>,=,!= operators, we cannot always be sure about correctness of the result. Thats why we use other techniques to compare doubles, for example, we can compare two doubles a and b by testing if their…
Rontogiannis Aristofanis
- 8,883
- 8
- 41
- 58
5
votes
1 answer
Minimize floating point inaccuracy in exponential moving average
Generally the formula is given as:
Davg, k = a * Davg, k – 1 + (1 – a) * Dk – 1
but while implementing it, if I do it as, just to save one floating point op,
Davg, k = a * ( Davg, k – 1 - Dk – 1 ) + Dk – 1
How much does it affect precision ? or is…
avd
- 13,993
- 32
- 78
- 99
5
votes
2 answers
How do I remove matrices from a list that are duplicates within floating-point error?
This question is similar to questions that have been asked regarding floating-point error in other languages (for example here), however I haven't found a satisfactory solution.
I'm working on a project that involves investigating matrices that…
Daniel Watkins
- 1,631
- 14
- 16
5
votes
5 answers
Why the digits after decimal are all zero?
I want to perform some calculations and I want the result correct up to some decimal places, say 12.
So I wrote a sample:
#define PI 3.1415926535897932384626433832795028841971693993751
double d, k, h;
k = 999999/(2*PI);
h = 999999;
d =…
Sunny
- 423
- 1
- 3
- 13
5
votes
1 answer
Floating point accuracy again
Yesterday I asked a question about why I was losing accuracy in a floating point arithmetic. I received an answer about how it was due to intermediate results being held in x87 registers. This was helpful but some of the details are still escaping…
john
- 85,011
- 4
- 57
- 81
5
votes
3 answers
float vs double (in Java)
Is there ever a case where these two methods would return different values given the same inputs?
int compare1(float a, float b)
{
return Double.compare(a, b);
}
int compare2(float a, float b)
{
return Float.compare(a, b);
}
By the same…
user1508893
- 9,223
- 14
- 45
- 57
5
votes
1 answer
To what precision does perl print floating-point numbers?
my $num = log(1_000_000) / log(10);
print "num: $num\n";
print "int(num): " . int($num) . "\n";
print "sprintf(num): " . sprintf("%0.16f", $num) . "\n";
produces:
num: 6
int(num): 5
sprintf(num): 5.9999999999999991
To what precision does perl…
garrett
- 207
- 1
- 5
- 12
5
votes
4 answers
Number precision error in C
Here is a code I wrote:
#include
#include
int main()
{
double num;
int tmp;
printf("enter a number!\n");
scanf("%lf",&num);
tmp=num*10000;
printf(" temp=%d\n",tmp);
return 0;
}
When I enter the…
Rona Hirsch
- 71
- 1
- 2
- 4
5
votes
4 answers
Understanding floating point precision
Is it the case that:
Representable floating point values are densest in the real number line near zero?
Representable floating point values grow sparser (exponentially?) as the number line moves away from zero?
If the above two are true, does that…
Chris Dargis
- 5,891
- 4
- 39
- 63