Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
0
votes
2 answers

Taking Logarithms of relatively small numbers in different languages/architectures/operating systems

In Java I run: System.out.println(Math.log(249.0/251.0)); Output: -0.008000042667076265 In C# I run: <- fixed Math.Log (x/y); \\where x, y are almost assuredly 249.0 and 251.0 respectively Output: -0.175281838 (printed out later in the…
Ritwik Bose
  • 5,889
  • 8
  • 32
  • 43
0
votes
2 answers

Strict inequality for floating point multiplication

Suppose a, x, and y are positive IEEE floating point numbers with x < y. Prove that a×x < a×y where × denotes floating-point multiplication rounding to nearest. Naively, you might suppose that for some a and for x close to y, you would get a×x =…
cffk
  • 1,839
  • 1
  • 12
  • 17
0
votes
1 answer

CPU vs GPU floating point precision of the MVP matrix

I have a problem that I believe is due to floating point errors on the CPU. I'm currently working on shadowmaps and at first I had the MVP calculations on the GPU e.g layout(location = 0) in vec3 inPos; uniform mat4 projectionMatrix; uniform mat4…
Edvin
  • 1,841
  • 4
  • 16
  • 23
0
votes
1 answer

OpenGL: glReadPixels inaccurate for unprojection

I have my own unproject function for performing reverse projection of a screen point. The code is as follows (written in OpenTK): public static Vector3 UnProject(Point screenLocation, float depth) { int[] viewport = GetViewport(); Vector4…
Maghoumi
  • 3,295
  • 3
  • 33
  • 49
0
votes
1 answer

Splitting a floating point number as sums of floating point of fixed precision

Suppose i have an algorithm by which i can compute an infinitely precise floating point number (depending from a parameter N) lets say in pseudocode: arbitrary_precision_float f = computeValue(n); //it could be a function which compute a specific…
user8469759
  • 2,522
  • 6
  • 26
  • 50
0
votes
1 answer

Doing calculation with NSDecimalNumber or Float

I have to do calculation in my application, i have formules like this one : result = Capital * rate / (1- 1/(1+ rate)^frequence) I have read in internet that doing calculation with floats can be lossless. Should i use NSDecimalNumber in my…
samir
  • 4,501
  • 6
  • 49
  • 76
0
votes
0 answers

denormal flushing in floating point arithmetic, function approximation

Out of curiosity, what's the meaning of "denormal flushing" in the context of floating point arithmetic? Do you know a good reference for such topic? I have some book but actually i can't find anything. I need to understand how this definition is…
user8469759
  • 2,522
  • 6
  • 26
  • 50
0
votes
3 answers

Dividing a double with integer

I am facing an issue while dividing a double with an int. Code snippet is : double db = 10; int fac = 100; double res = db / fac; The value of res is 0.10000000000000001 instead of 0.10. Does anyone know what is the reason for this? I am…
hardcoder
  • 21
  • 1
  • 1
  • 1
0
votes
0 answers

How to check if the sum of floats equals a number and accounting for Floating Point Inaccuracy

I'm trying to check if a sum of floating point numbers adds up to 100. And of course they don't, due to floating point accuracy. So, what doesn't work is: test = [99.9, 0.025, 0.025, 0.0125, 0.0125, 0.025] sum(test) == 100. alternatively sum(test)…
MichaelA
  • 1,866
  • 2
  • 23
  • 38
0
votes
3 answers

comparing GPS cordinates with a certain margin in c

I have been tying to make a device with arduino that would use two gps modules. So one sends gps cords and the other receives and compares its current location to the location of the sender and once the receiver gets a few meters from the sender an…
Ismar Perez
  • 19
  • 1
  • 5
0
votes
0 answers

Ways to Verify Floating Point Operations

NOTE: First let me be clear that this question is related to a side project I've been working on for fun. This is NOT going to be used in any production/professional code. So answers such as "why did you do this your self when you could have just…
Mandalf The Beige
  • 81
  • 1
  • 8
0
votes
2 answers

Why does 99.99 / 100 = 0.9998999999999999

Possible Duplicate: Dealing with accuracy problems in floating-point numbers Whereas 99.99 * 0.01 = 0.99 Clearly this is the age old floating point rounding issue, however the rounding error in this case seems quite large to me; what I mean is I…
redcalx
  • 8,177
  • 4
  • 56
  • 105
0
votes
1 answer

Browser wrong javascript float calculation?

If you put this in the browser address bar (firefox/IE8): javascript:alert(40.34+56.87) Why the result comes as 97.2100000000001 when its supposed to be 97.21
JavaSheriff
  • 7,074
  • 20
  • 89
  • 159
0
votes
3 answers

Why `0.4/2` equals to `0.2` meanwhile `0.6/3` equals to `0.19999999999999998` in python?

I know these are float point division. But why did these two formula behave differently? And I did some more investigation, the result confusing me even more: >>>0.9/3 0.3 >>>1.2/3 0.39999999999999997 >>>1.5/3 0.5 What's the logic here to decide…
Zen
  • 4,381
  • 5
  • 29
  • 56
0
votes
2 answers

Floating point number association error

When I do a floating point addition I get different results. My database is 32 bit Kognitio. Can some one explain me better why this is a problem when I have my floating point values well within the limits. I do understand that the operations…
Srini V
  • 11,045
  • 14
  • 66
  • 89
1 2 3
…
90 91