Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
4
votes
3 answers
Midpoint 'rounding' when dealing with large numbers?
So I was trying to understand JavaScript's behavior when dealing with large numbers. Consider the following (tested in Firefox and Chrome):
console.log(9007199254740993) // 9007199254740992
console.log(9007199254740994) //…
p.s.w.g
- 146,324
- 30
- 291
- 331
4
votes
4 answers
Converting float to UInt32 - which expression is more precise
I have a number float x which should be in <0,1> range but it undergo several numerical operations - the result may be slightly outside <0,1>.
I need to convert this result to uint y using entire range of UInt32. Of course, I need to clamp x in the…
Libor
- 3,285
- 1
- 33
- 41
4
votes
1 answer
Linq Sum() precision
In my project I have been using Linq's Sum() a lot. It's powered by NHibernate on MySQL. In my Session Factory I have explicitly asked NHibernate to deal with exactly 8 decimal places when it comes to decimals:
public class DecimalsConvention :…
Doug Peters
- 529
- 2
- 8
- 17
4
votes
1 answer
assertions on negations of relational operators for floating point number
Suppose I have two variables a and b, either both in type float, or both in type double, which hold some values. Do the following assertions always hold? I mean, do the existence of numerical errors alter the conclusions?
a > b is true if and only…
shaoyl85
- 1,854
- 18
- 30
4
votes
4 answers
General floating-point maths query
Okay so I get that some numbers can't be represented properly in binary just like 1/3 can't be fully represented in decimal.
So how come when I console.log(0.3) it returns 0.3
but when I console.log(0.1 + 0.2) it returns the 0.30000000000000004
How…
robjtede
- 736
- 5
- 16
4
votes
5 answers
working decimal point numbers
I wrote a program in java about floating point but the output obtained is not an accurate one; whereas the same program written in C did produce an accurate result.
Code Snippet:
public class One {
public static void main(String...args){
…
user3086472
- 41
- 1
4
votes
1 answer
Math.Tan() near -Pi/2 wrong in .NET, right in Java?
I have an unit test failing on a Math.Tan(-PI/2) returning the wrong version in .NET.
The 'expected' value is taken from Wolfram online (using the spelled-out constant for -Pi/2).
See for yourselves here.
As correctly observed in the comments, the…
Cristian Diaconescu
- 34,633
- 32
- 143
- 233
4
votes
1 answer
Good tolerance for double comparison
I am trying to come up with a good tolerance when comparing doubles in unit tests.
If I allow a fixed tolerance as I've seen mentioned on this site (eg return abs(actual-expected) < 0.00001;), this will frequently fail when numbers are very big…
Jon Brooks
- 2,472
- 24
- 32
4
votes
5 answers
different behaviours for double precision on different compiler
my code is quite simple
double d = 405, g = 9.8, v = 63;
double r = d * g / (v * v);
printf("%s\n",(r>1.0)?"GT":"LE");
and here is my result
g++-mingw32-v4.8.1: LE (the result is EQ indeed)
g++ on ubuntu : GT ( this result comes from my friend,…
sray
- 93
- 1
- 9
4
votes
1 answer
Why does g++ (4.6 and 4.7) promote the result of this division to a double? Can I stop it?
I was writing some templated code to benchmark a numeric algorithm using both floats and doubles, in order to compare against a GPU implementation.
I discovered that my floating point code was slower and after investigating using Vtune Amplifier…
amckinley
- 629
- 1
- 7
- 15
4
votes
2 answers
floating point operations in go
Here's the sample code in go:
package main
import "fmt"
func mult32(a, b float32) float32 { return a*b }
func mult64(a, b float64) float64 { return a*b }
func main() {
fmt.Println(3*4.3) // A1, 12.9
fmt.Println(mult32(3,…
shycha
- 440
- 5
- 13
4
votes
4 answers
How to actually avoid floating point errors when you need to use float?
I am trying to affect the translation of a 3D model using some UI buttons to shift the position by 0.1 or -0.1.
My model position is a three dimensional float so simply adding 0.1f to one of the values causes obvious rounding errors. While I can use…
Caustic
- 377
- 1
- 2
- 9
4
votes
3 answers
Why does scipy.stats.nanmean give different result from numpy.nansum?
>>> import numpy as np
>>> from scipy import stats
>>> a = np.r_[1., 2., np.nan, 4., 5.]
>>> stats.nanmean(a)
2.9999999999999996
>>> np.nansum(a)/np.sum(~np.isnan(a))
3.0
I'm aware of the limitation of floating point representation. Just curious…
herrlich10
- 6,212
- 5
- 31
- 35
4
votes
1 answer
What is gnuplot's internal representation of floating point numbers?
I'm sure the answer is obvious but I can't find it without looking at the source.
What is gnuplot's internal representation of floating point numbers? Is it the platform's double? Does it use its own internal representation? Can it do arbitrary…
iter
- 4,171
- 8
- 35
- 59
4
votes
4 answers
float - c data types
int main() {
float a = 20000000;
float b = 1;
float c = a+b;
if (c==a) { printf("equal"); }
else { printf("not equal");}
return 0;
}
when I run this it says "equal".
but when I change the value of a to 2000000 (one zero less) the answer is…
wantToLearn
- 481
- 4
- 7
- 19