Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
6
votes
4 answers
Is it always necessary to use float literals when performing arithmetic on float variables in C++?
I see a lot of C++ code that has lines like:
float a = 2;
float x = a + 1.0f;
float b = 3.0f * a;
float c = 2.0f * (1.0f - a);
Are these .0f after these literals really necessary? Would you lose numeric accuracy if you omit these?
I thought you…
Joan Venge
- 315,713
- 212
- 479
- 689
5
votes
3 answers
Does the dot in the end of a float suggest lack of precision?
When I debug my software in VS C++ by stepping the code I notice that some float calculations show up as a number with a trailing dot, i.e.:
1232432.
One operation that lead up to this result is this:
float result = pow(10, a * 0.1f) / b
where a…
Thomas Johansson
- 583
- 2
- 6
- 15
5
votes
3 answers
Exact binary representation of a double
Possible Duplicate:
Float to binary in C++
I have a very small double var, and when I print it I get -0. (using C++).
Now in order to get better precision I tried using
cout.precision(18); \\i think 18 is the max precision i can…
user690936
- 985
- 5
- 13
- 23
5
votes
3 answers
Machine precision estimations
Some people say that machine epsilon for double precision floating point numbers is 2^-53 and other (more commonly) say its 2^-52. I have messed around estimating machine precision using integers besides 1 and aproaching from above and below (in…
njvb
- 1,377
- 3
- 18
- 36
5
votes
2 answers
What is the relationship between digits of significance and precision loss in floating point numbers?
So I have been trying to wrap by head around the relation between the number of significant digits in a floating point number and the relative loss of precision, but I just can't seem to make sense of it. I was reading an article earlier that said…
MoarCodePlz
- 5,086
- 2
- 25
- 31
5
votes
1 answer
5
votes
2 answers
Avoiding Possible Precision Loss with a Simple Moving Average
Suppose we had a basic moving average function that was keeping track of the sum. For example:
Queue values;
double sum;
double CalcSMA(double next) {
values.push(next);
sum -= values.pop();
sum += next;
return sum /…
Jay
- 983
- 2
- 8
- 23
5
votes
3 answers
Python: Float infinite length (Precision float)
My code:
def calc_pi(acc):
pos = False
sum = 4.0
for i in range(2, acc):
if not pos:
sum -= 4.0/(2*i-1)
pos = True
else:
sum += 4.0/(2*i-1)
pos = False
…
d0n.key
- 1,318
- 2
- 18
- 39
5
votes
1 answer
Can an executable in Linux influence the floating point precision within a linked dynamic library?
We are experiencing an issue with floating point precision within a dynamic library.
The set-up is as follows:
We have a dynamic library, which performs a computation X on a large array of floating point numbers. X consists of a lot of floating…
Coert Metz
- 894
- 6
- 21
5
votes
1 answer
Given a number, how to find a closest number in a series of floating point data
I came across this question while looking for Amazon interview questions and wanted to ask.
Given a number, how can you find the closest number in a series of floating point data?
If everything is integer , answer is substracting the number from…
Dogacan Sbdb
- 69
- 4
5
votes
3 answers
Is it possible to get floating point error in this case?
I know that float arithmetic is tricky, but I am not sure if it is possible to get a mistake if you have a division followed by the inverse multiplication. Written in code, is it possible, that this method will return false:
public boolean…
Ivaylo Toskov
- 3,911
- 3
- 32
- 48
5
votes
2 answers
Why isn't to!int() working properly?
Why does this assertion fail?
import std.conv;
void main()
{
auto y = 0.6, delta=0.1;
auto r = to!int(y/delta);
assert(r == 6);
}
r's value should be 6 and yet it's 5, Why?
Algo
- 841
- 1
- 14
- 26
5
votes
1 answer
Fix floating point imprecision in ceiling
The problem:
ceiling(31)
#31
ceiling(31/60*60)
#32
What is the correct way to fix this kind of errors?
Doing the multiplication before the division is not an option, my code looks something like this:
x <- 31/60
...
y <- ceiling(x*60)
I'm…
pomber
- 23,132
- 10
- 81
- 94
5
votes
2 answers
PHP. result of the subtraction of two floating point numbers
For example...
$aa = 10694994.89;
$bb = 10696193.86;
$ab = $aa - $bb;
// result is:-1198.9699999988 not the -1198,97
But in this exampe:
$cc = 0.89;
$dd = 0.86;
$cd = $cc - $dd;
//Result is: 0.03
Why the difference in to examples? Lacks…
Andrey K.
- 655
- 1
- 8
- 16
5
votes
1 answer
Finding next IEEE 754 representable number (towards -INF) with C?
I am trying to write a function that takes in a 32-bit floating point number (that has been converted from a 32-bit binary string) and returns the previous representable float in 32-bit binary. So far I have the conversion from binary to float down,…
user3614396
- 53
- 4