Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
0
votes
1 answer

Difference in sum of an array using python and matlab code

Here is the link to the mat file temp whose sum i want. The sum for python is 1.1230325644146074e-10. The sum in matlab is 1.2189e-010. But when i copy the array from matlab and add them in python terminal i get the same sum as that of matlab. What…
user38751
  • 1
0
votes
2 answers

accurate image aspect ratio in python

I have the following code to get the aspect ratio of an image img0 = color.rgb2gray(io.imread("C:\\work\\TRAIN\\SET1\\bus.jpg")) img0 = resize(img0, (40, 116)) ar = 1.0 * (img0.shape[1]/img0.shape[0]) print "aspect ratio: " print ar The output is…
user961627
  • 12,379
  • 42
  • 136
  • 210
0
votes
4 answers

Frac function losing precision

I have a TDateTime variable which is assigned a value at runtime of 40510.416667. When I extract the time to a TTime type variable using the Frac function, it sets it to 0.41666666666. Why has it changed the precision of the value and is there a…
PDM
  • 503
  • 2
  • 12
  • 27
0
votes
1 answer

How can I do conditional compilation branching based on the real type used?

I was doing a solution for this question with respect to underlying hardware and platform. Due Delphi's code generator specifics, it produces 80x87 instructions for IA32 platform and SSE instructions for AMD64 platform, thus having generic Real…
Free Consulting
  • 4,300
  • 1
  • 29
  • 50
0
votes
3 answers

How does casting from float to double work in C++?

The mantissa bits in a float variable are 23 in total while in a double variable 53. This means that the digits that can be represented precisely by a float variable are log10(2^23) = 6.92368990027 = 6 in total and by a double variable log10(2^53)…
ksm001
  • 3,772
  • 10
  • 36
  • 57
0
votes
3 answers

precision error in nth root of a number in C++

I know from previous threads on this topic that using float arithmetic causes precision anomalies. But Interestingly I observed that the same function is behaving in two different ways.Using COUT output is 4 but if I am saving the result into a…
vaibhavatul47
  • 2,766
  • 4
  • 29
  • 42
0
votes
1 answer

Floating point arithmetics using to nearest

Concerning the library fenv.h in C, I have some difficulties to understand what the rounding direction mode does when using fesetround() and the direction FE_TONEAREST (default) in particular. There are 4 types of rounding directions in the library…
awryl
  • 28
  • 2
0
votes
1 answer

Increment/decrement to floating number seem off

I have a float to represent the zoom factor on an image. setZoomPercent( currentZoomPercent - 0.1f ); The trouble I'm having is that the decrementation is giving the following result. How can I avoid this…
James P.
  • 19,313
  • 27
  • 97
  • 155
0
votes
0 answers

Calculate the difference between 2 double numbers

Why do I get 0.6100000000000001 when I should get 0.61 ? I know the get I know the decimal format to fix it. I just want the explanation for why it happens. public class Mid2009{ public static void main (String args []){ double d1 =…
Mahmoud Nabegh
  • 1
  • 2
0
votes
1 answer

C++ rotation of point in float coordinates accuracy

i study OpenGL ES 2.0. But i think it's more C++ question rather then OpenGL. I'am stuck with rotation question. It is known, that rotation transformation can be applied using the following equations: p'x = cos(theta) * (px-ox) - sin(theta) *…
Alex Hoppus
  • 3,821
  • 4
  • 28
  • 47
0
votes
2 answers

One minus a small number equals one? Dealing with small numbers

The title is self explanatory. What is going on here? How can I get this not to happen? Do I really have to change all of my units (it's a physics problem) just so that I can get a big enough answer that python doesn't round 1-x to 1? code: import…
doublefelix
  • 952
  • 1
  • 9
  • 22
0
votes
3 answers

strange arithmetic with swi-prolog

I find the result very strange. Why not 0.3? Can somebody tell me why this result? Is it possible to fix this. ?- X is 5.3-5. X = 0.2999999999999998. ?- My second question is how would I transform from 'hour' notation '13.45'…
vincent
  • 101
  • 1
  • 9
0
votes
1 answer

Cases where a classifier fails/performs good

I am using Naive Bayes to classify a set of documents and I was wondering if it possible to get concrete examples of where the classifier does well/fails on the test set. I am measuring accuracy, precision and recall, but in my analysis I can only…
Crista23
  • 3,203
  • 9
  • 47
  • 60
0
votes
1 answer

Calculate square root for a number that is two floating numbers' difference

I am working on floating point calculation by C# on Win7. double a, b; double diff = a -b; because of floating point precision, sometimes a - b is a very small non-0 number even though a == b. but, sometimes, a and b are close but they are not…
user2420472
  • 1,127
  • 1
  • 12
  • 20
0
votes
2 answers

C# floating point number's precision of display and calculation, which are different

I am doing calculation by C# on Win7. I got a double number precision problem. double x = 3389774.583; string ss = "x is " + x + " x square is " + x * x + " , 11490571723552.8 - x*x is " + (11490571723552.8 - x * x) + "\n"; …
user2420472
  • 1,127
  • 1
  • 12
  • 20
1 2 3
…
90 91