Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
0
votes
2 answers

Is floating point division sufficient to determine bin number accurately?

I'm trying to account for crossings of the prime meridian accurately and I ran into the following question about IEEE floating point arithmetic (round to nearest): Let n be an integer, and d a small positive number. Does y = n * 360 - d < n *…
cffk
  • 1,839
  • 1
  • 12
  • 17
0
votes
2 answers

Accuracy in the sum of an array of NSNumber

I have an array of NSNumber (in this case, for simplicity I have only one element) The array is "_incassiInDistinta" that contains an object "Incasso" which has a property "_importoIncassato" Now if I use this double sum = [[_incassiInDistinta…
matti157
  • 1,288
  • 2
  • 13
  • 26
0
votes
2 answers

if statement not working in MATLAB

I'm trying to create a Euler approximation function in MATLAB. I am supposed to evaluate at t = 0.1, 0.2, 0.3, 0.4 but for some reason my code does not evaluate at 0.3. Here is my code: clear; clc; stepSizes = [0.1 0.05 0.025]; tInitial =…
user3642365
  • 549
  • 1
  • 6
  • 16
0
votes
2 answers

Fixed Point to Floating Point and Backwards

Is converting Fixed Pt. (fixed n bit for fraction) to IEEE double safe ? ie: does IEEE double format can represent all numbers a fixed point can represent ? The test: a number goes to floating pt format then back to it's original fixed pt format.
Ahmed Khalaf
  • 1,220
  • 12
  • 28
0
votes
2 answers

Wierd result in formatting float value to string using String.format

I'm kinda recent to Java. I understand in general the issues around floating point precision and conversion but not sure why am seeing such a ridiculous result in Java... I did this... String str1 = String.format("%.02f", 0.001921921f); String str2…
dangling_refrenz
  • 325
  • 3
  • 12
0
votes
1 answer

Scheduling events when time is a floating point value

The following example highlights a pitfall with regard to the use of floating point numbers: available_again = 0 for i in range(0,15): time = 0.1*i if time < available_again: print("failed to schedule at " + str(time)) …
Baz
  • 12,713
  • 38
  • 145
  • 268
0
votes
1 answer

Open GL pixel-perfect rendering blurred in MacOSX

I'm writing a cross-platform application which draws a 2d image read from PNG file using OpenGL. My program works on Linux, Windows and Mac OS X. In Linux and Windows it works fine, but in Mac OS X the image is not drawn pixel-perfect. The image…
igagis
  • 1,959
  • 1
  • 17
  • 27
0
votes
0 answers

Improving accuracy of Gauss-Hermite weight-calculating algorithm

I am writing a parallelized version of the algorithm to calculate the weights and abscissae for the Gauss-Hermite quadrature (details of which can be found here or here). I have the following algorithm to compute the values using GPU…
Thomas Russell
  • 5,870
  • 4
  • 33
  • 68
0
votes
2 answers

JavaScript: Downsides of toFixed(...) method

In my programs, I need "float" numbers rounded to the nearest two decimal digits, and after some research I decided to use toFixed(..) for this purpose, like shown in the example below. What are the downsides of using toFixed()? Does it work in all…
FranXh
  • 4,481
  • 20
  • 59
  • 78
0
votes
0 answers

Precision error with logarithms and exponential functions in Ruby

I tried this: t = (Math.log10(7)+3)/2 10.0**(2.0*t-3.0) => 7.000000000000002 The result should be 7 not 7.000000000000002. Why?
rtacconi
  • 14,317
  • 20
  • 66
  • 84
0
votes
0 answers

Controlling accuracy while iterating chaotic maps in Matlab

I'd like to be able to control the decimal accuracy exactly while iterating a chaotic map. In particular, I'd like to make sure that Matlab uses no more / no less than the value p I specify in the following function: function out =…
caseyalan
  • 3
  • 3
0
votes
0 answers

What could cause a program to be less accurate in C than in matlab?

I wrote an orbit propagator (calculates position and velocity of a satellite for each time step) in matlab and it's pretty accurate (4.5 km in a 24-hour propagation). I had to rewrite it in C, I did it and it's working properly but my results are…
la_
  • 51
  • 5
0
votes
1 answer

Efficiently counter floating point number error

I'm having a problem in both Java and MATLAB where two numbers which should be the same apparently aren't. Java: float f1 = 0.3f-0.2f; float f2 = 0.4f-0.3f; System.out.println(f1==f2); // Prints false System.out.println(f1); // Prints…
Eduardo
  • 6,900
  • 17
  • 77
  • 121
0
votes
3 answers

Reading string in to float produces incorrect accuracy

Now I understand floats are less accurate than double, but does this explain when I have the std::string: "7.6317" and I do: float x = atof(myString.c_str()); getting 7.63170004 is expected? Is there any way I can tell the assignment of x to only…
user997112
  • 29,025
  • 43
  • 182
  • 361
0
votes
2 answers

why does JavaScript mess up 0.1 + 0.2 when C++ doesn't?

I understand that with the IEEE representation (or any binary representation) for double one can't represent 0.1 with a finite amount of bits. I have two questions with this in mind: When C++ also uses the same standard for double why doesn't it…
batman
  • 5,022
  • 11
  • 52
  • 82
1 2 3
…
90 91