Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
0
votes
1 answer

Using GREATEST() on float type values gives inaccurate results

I was experimenting with mysql, and made a query to compare two different fields using the GRATEST() function. My query looks like this: SELECT id, float1, float2, GREATEST( IFNULL(float1, 0), IFNULL(float2, 0) ) AS gtst FROM…
Gergely Lukacsy
  • 2,881
  • 2
  • 23
  • 28
0
votes
2 answers

Upper bound for number of digits of big integer in different base

I want to create a big integer from string representation and to do that efficiently I need an upper bound on the number of digits in the target base to avoid reallocating memory. Example: A 640 bit number has 640 digits in base 2, but only ten…
John
  • 515
  • 4
  • 11
0
votes
3 answers

MySql and float or double crop my number when I update it per query from php

I like to store Latitudes and Longitudes in a very precise way into my MySql Database with InnoDB. However, float did not offer enough internal decimal places so I switched to double. Wondering myself a little but MySql accepted double with a size…
TheMAn
  • 71
  • 1
  • 10
0
votes
1 answer

Arithmetic error with double c++

I have noticed a small error on some arithmetic calculations using double. It is really weird, there's always a small error and/or an extra significant digit. First I am using atof to convert a number that has two significant digits that I am…
Luis Cruz
  • 1,488
  • 3
  • 22
  • 50
0
votes
1 answer

Is Double Math Consistent across Multiple Platforms?

For my deterministic physics engine, I need to confirm that calculations with doubles in C# are consistent enough across multiple platforms. Does anyone know how much the following functions differ in results? On my computer as a Windows 32 bit…
JPtheK9
  • 275
  • 2
  • 12
0
votes
2 answers

Float precision

Due to precision of the microcontroller, I defined a symbol containing ratio of two flotants numbers, instead of writing the result directly. #define INTERVAL (0.01F/0.499F) instead of #define INTERVAL 0.02004008016032064F But the first…
Stack Over
  • 425
  • 2
  • 8
  • 18
0
votes
3 answers

Java error on simple multiplication

I'm currently in the process of learning Java. I have just run the following line in Eclipse in Win7 using jre1.8.0_25: System.out.println(4.5 * 7.9); The console output is: 35.550000000000004 I'm just wondering why the output is wrong.
desiguel
  • 505
  • 3
  • 10
0
votes
1 answer

Adding multiple Numbers with decimal shows long trail of numbers

I am trying to add few decimal numbers from form input field which is like: var total = Number(11000.2)+Number(10000.1)+Number(10762.4); consol.log(total); consol shows 31762.700000000004 I don't understand from where its getting addition values…
sarojanand
  • 607
  • 1
  • 11
  • 30
0
votes
2 answers

Accuracy of Rosenbrock's test function calculation

I want to calculate Rosenbrock's test function I have implemented the following C/C++ code #include /********/ /* MAIN */ /********/ int main() { const int N = 900000; float *x = (float *)malloc(N * sizeof(float)); for (int…
Vitality
  • 20,705
  • 4
  • 108
  • 146
0
votes
2 answers

Floating Number in C

Today I was playing with floating-point number in C code likes: float a = 0.0; a += 0.8; if(a == 0.800000) printf("correct"); The if statement do not get executed and for this I printed the value of a which was 0.800000. void main(){ float…
Asis
  • 683
  • 3
  • 23
0
votes
1 answer

How to calculate expressions in better accuracy

I tried to calculate this: limit(x^5−6*x^4+14*x^3−20*x^2+24*x−16, x, 1.999993580023622); But I got 0. I think it happens because a loss of significance. What can I do to get more precise result?
Asaf Kahlon
  • 104
  • 1
  • 9
0
votes
3 answers

Comparing a float and an int

Due to safety reasons, I need to perform the same computation twice, one time with only integer (int32) variables and another time with only float (float32) variables. At the end of the computation a comparison between the two results is taking…
Eagle
  • 3,362
  • 5
  • 34
  • 46
0
votes
5 answers

How do I check accuracy on specified decimal places in C

I need to specify that I need two numbers be the same upon 3rd decimal place: 1.2345 and 1.2348 is correct. But 1.2345 and 1.2388 is not correct. And I need let user specify how many places should program check. I was thinking about something like…
sczdavos
  • 2,035
  • 11
  • 37
  • 71
0
votes
0 answers

Using the bcmath functions correctly

I've never used the bcmath figures before and having been doing a bit of reading about comparing and manipulating floating point numbers and this has led me to the bcmath functions. I understand you definitely should use something like the bccomp…
Brett
  • 19,449
  • 54
  • 157
  • 290
0
votes
2 answers

Is there a way to distinguish integers from very near decimals in Javascript?

Look at those evaluations (actual dump from node 0.10.33) > parseFloat(2.1e-17) === parseInt(2.1e-17) false > parseFloat(2.1e-17 + 2) === parseInt(2.1e-17 + 2) true > parseFloat(2.000000000000000000000000000000000009) ===…
seldon
  • 977
  • 2
  • 8
  • 20
1 2 3
…
90 91