Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
-1
votes
1 answer

truncate the elements of a float array

This might be a strange question, but how do i truncatean array of floats to e.g.: 0.2f (2 decimal places). Imagine i have a float array X that contains 10 elements with the value 2.123456. I want to make them 2.123. Can anybody help? Kind regards
Pedro Neves
  • 364
  • 1
  • 8
  • 26
-1
votes
2 answers

Incrementing value in a loop yields different (wrong) value?

I have spent almost all day trying to figure out why this is happening, I had code that incremented a float if it was not a whole number until it was a whole number, i'd trigger it by incrementing the value by 0.01 and the loop keeps incrementing it…
Lee Fogg
  • 775
  • 6
  • 22
-1
votes
2 answers

What implementation of average is the most accurate?

Given those two implementations of the average function: float average(const vector& seq) { float sum = 0.0f; for (auto&& value : seq) { sum += value; } return sum / seq.size(); } And: float average(const vector&…
ereOn
  • 53,676
  • 39
  • 161
  • 238
-1
votes
2 answers

int*float accuracy

I know about differences between number representations but here is something that I would like someone to explain. we see that value of float t is as it is, and I wonder why f is not equal to 2+t (as mathematically should be) but there is that…
4pie0
  • 29,204
  • 9
  • 82
  • 118
-1
votes
1 answer

R Showing Different Results in forward and backward for loop

I get different answers while searching for a certain element in an array if I am searching forward or backwards using a for loop. Example: Code that gives CORRECT ANSWER vg = rep(seq(0.9,1.1,0.01),90) vals = seq(0.9,1.05,0.01) for(val in…
Amit
  • 41
  • 6
-2
votes
1 answer

Why is the result of this floating-point calculation not consistent?

Why is the answer of this code snippet 400.002 and not just 400? #include int main(){ using namespace std; float a = 123.4; a = (a - int(a))*1000; cout << a << endl; return 0; } I'm a newbie. Is this due to the…
Bad Athon
  • 1
-2
votes
1 answer

Java (suggestion) - implement double as two integers?

In java, double takes 64 bits, but stores (or computes with) numbers unprecisely. E.g. the following code: double a = 10.125d; double b = 7.065d; System.out.println(a-b); prints out 3.0599999999999996 rather than 3.06. So, the question - what…
milkamar
  • 443
  • 6
  • 16
-2
votes
1 answer

In swift is there a difference between a double and a float?

I wrote a basic program for calculating money. If I switch the data type of the functions to a float. The count for pennies is incorrect if I set my initial value to 16.16. If I switch it to a double it corrects it. I am not sure why this is…
Shawn Sandhu
  • 1
-2
votes
1 answer

precision errors with numpy

Can someone explain this: >>> numpy.array([22.0], dtype=numpy.float64) / (2 ** 11) array([0.01074219]) >>> 22.0 / (2 ** 11) 0.0107421875 >>> Numpy seems to generates an erroneous result, probably some kind of precision error. How can I fix this?
Julien REINAULD
  • 599
  • 2
  • 5
  • 18
-2
votes
2 answers

How to achieve maximal accuracy in the sum of series calculation?

I wrote a code to calculate the sum of the series 2^(-k), but I don't know how to improve the accuracy of this calculation. This is what I've done so far. #include #include using namespace std; int main() { int i, n; …
kamacite
  • 11
  • 2
-2
votes
1 answer

MySQL Returns Wrong Value Only When Number Is Larger

I am writing a bot that has stored values. One of which is a number of 18 places. I am able to store the values properly into my database, MySQL, as a BIGINT[20]. The values being stored are correct. The problem comes when trying to fetch that…
Vince
  • 23
  • 2
-2
votes
1 answer

Float subtraction gives me inaccurate results

Im writing a code where measurements, that are taken every half second, need to be subtracted from an initial value to reach 0 eventually. Both values are float. Initial value is 140 000 000 and measurements range from 0.320000001 to 0.389999999. …
Inzzza8
  • 17
  • 1
  • 4
-2
votes
1 answer

Why 4.7 is 4.6999999 in c

' #include #include int main(){ int i; float num = 4.700; for(i=1;i<5;i++){ printf("%0.3f\tx%d\t\t=%d\n",num,(int)pow(10,i),(int)(num*pow(10,i))); } return…
Devan
  • 87
  • 1
  • 1
  • 6
-2
votes
1 answer

C# tracking precision and accumulated rounding error in number types

I am writing some code to do some math (for a research project) I am not sure what level of precision I am going to need, or how much of a difference rounding errors could introduce in my results. For example one thing I want to do is calculate the…
user802599
  • 787
  • 2
  • 12
  • 35
-2
votes
2 answers

What's the accuracy difference between the two versions derivative calculation?

The following is Python code aiming to calculate the derivative of a given function f. Version one (Solution) x[ix] += h # increment by h fxh = f(x) # evalute f(x + h) x[ix] -= 2 * h fxnh = f(x) x[ix] += h numgrad = (fxh - fxnh) / 2 / h Version…
GabrielChu
  • 6,026
  • 10
  • 27
  • 42
1 2 3
…
90 91