Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
-1
votes
3 answers

When does floating point round exactly?

My question was labeled as a duplicate of https://stackoverflow.com/a/22023918/856090 but it is not a duplicate. That question covers the case of inexact representation, while this my question is about exact representation in a special case In a…
porton
  • 5,214
  • 11
  • 47
  • 95
-1
votes
1 answer

If I reduce the precision of float to 8 will I get correct numbers when working with currency?

I know that if I use float numbers when calculating money, I will have the risk of getting incorrect results if the results are rounded to lower precision, isn't the results equal? >>> 234042163/(2**24) 13.949999988079071 >>>…
gawry
  • 762
  • 1
  • 9
  • 18
-1
votes
1 answer

NSNumbers and it's value representation issue

Explanation Why this is not a duplicate of that pointed Q&A!: I encountered this problem in relation to Objective-C minor implementation and NSNumber type has varying floating point precision and I just needed to know what really happens and what…
Randika Vishman
  • 7,983
  • 3
  • 57
  • 80
-1
votes
1 answer

Print out a really small floating-point number

I have the following code: float test = 1/(10^(15)); std::cout << "test: " << test << std::endl; But I get "test: 0" as the value. I want "test: 0.000000000000001". How do I achieve this?
daemoner119
  • 113
  • 1
  • 9
-1
votes
1 answer

modulo function

Newbie in C#, trying to work out a simple calculation. float old x=300 float Distance=300 float pitch=0.8 int sign=1 new x= old x - (sign * (Distance % pitch) * 0.5 f) The value generated by program for new x is 299.6 (which I don't…
Rockstar
  • 35
  • 2
-1
votes
2 answers

Does printf() modify its parameters?

I was trying to see how different languages handle floating point numbers. I know that there are some inherent issues in floating point representation, which is why if you do 0.3 + 0.6 in Python, you get 0.899999 and not 0.9 However, these snippets…
Akshay Damle
  • 1,220
  • 3
  • 18
  • 31
-1
votes
2 answers

IEEE754 float point substraction precision lost

Here is the subtraction First number Decimal 3.0000002 Hexadecimal 0x4040001 Binary: Sign[0], Exponent[1000_0000], Mantissa[100_0000_0000_0000_0000_0001] substract second number: Decimal 3.000000 Hexadecimal 0x4040000 Binary: Sign[0],…
Shuaiyu Jiang
  • 239
  • 1
  • 3
  • 15
-1
votes
1 answer

Perl addition tacking on mystery decimals

I am running a perl script and basic addition operators seem to be not working. For my code print $total_charges," 1 \n"; print $payments," payments\n"; $total_charges+=$payments; print $total_charges," 2 \n"; For this code, the output is the…
mrfish
  • 64
  • 7
-1
votes
3 answers

Correct Way to Compare Floating-point Numbers

I'm calculating a real numeric value of the form N + fraction. Say, For example, N + fraction = 7.10987623, then N = 7 and fraction = 0.10987623 Next, I need to check to see if fraction is greater than or equal to the ratio 23269/25920. The…
JCM
  • 197
  • 1
  • 1
  • 10
-1
votes
1 answer

In java, 33.6f*100 != 3360

In Java (Eclipse), 33.6f*100 gives 3359.9998. Even if 33.6*100 gives 3360. I don't succeed to reproduce this issue with other values (for example 32.6f*100 gives 3260). Do you know what is it due to and how I can fix the issue? in my case 33.6f is…
johanvs
  • 4,273
  • 3
  • 24
  • 26
-1
votes
1 answer

Rounding values for normalised variables

I'm trying to write a function in Python which gives me values for normalized variables in a desired number of steps. I'm having a problem with rounding values and reaching 1. Since I will need these values in later computations, it is not good…
python starter
  • 183
  • 1
  • 12
-1
votes
2 answers

how to keep floating point precision in R?

I assign a floating point number to a variable in R. e.g. k <- 1200.0000002161584854 I got > k [1] 1200 > k+0.00000001 [1] 1200 How to keep the precision of k ? I have read some posts here, but, I do not find a solution.
user3440244
  • 371
  • 1
  • 3
  • 15
-1
votes
5 answers

Objective C, division between floats not giving an exact answer

Right now I have a line of code like this: float x = (([self.machine micSensitivity] - 0.0075f) / 0.00025f); Where [self.machine micSensitivity] is a float containing the value 0.010000 So, 0.01 - 0.0075 = 0.0025 0.0025 / 0.00025 = 10.0 But in…
Lucian Thorr
  • 1,997
  • 1
  • 21
  • 29
-1
votes
1 answer

Inconsistent floating point substraction

in Javascript: This expression (2.0 - 1.1) returns 0.8999999999999999. However this one (4.0 - 1.1) return 2.9. Can anybody explain this inconsistency? I understand that 1.1 can't be represented in floating point.
ngurahbaskara
  • 1
  • 1
-1
votes
1 answer

Different float input and output

float amount; printf("Enter the amount:\n"); scanf("%f", &amount); // input: 100.10 printf("%f", amount); Output: 100.099998 The problem is that the output is not 100.10 same as the input;
Naim Ibrahim
  • 266
  • 1
  • 2
  • 10
1 2 3
…
90 91