Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
0
votes
1 answer

From what number the following conversion integer -> double -> integer is not valid?

Assuming that my computer use IEEE 754 floating-point encoding, I wonder what is the smallest number for which the following function return false: constexpr bool test(const unsigned long long int x) { return static_cast
Vincent
  • 57,703
  • 61
  • 205
  • 388
0
votes
0 answers

Create NSArray with float value in a range of -1.0 +1.0 from C Array in identical format

Ok, maybe I'm tired but have some trouble when try to passing "float value" from an C array style to NSarray across NSNumber and obtain identical data. What I would like achieve is pass this value in other class for plotting audio…
sundsx
  • 588
  • 9
  • 27
0
votes
0 answers

why is 123.56 - 62.12 = 61.440000000000005

I am just starting with python and was running through a tutorial here Running the exmamples, I was surprised to find print("123.56 - 62.12 =", 123.56 - 62.12) To come out as 123.56 - 62.12 = 61.440000000000005 I am running python v3.3.0 Ran…
demiurg
  • 108
  • 10
0
votes
1 answer

Keeping accuracy when taking decimal to power of integer

My code is as follows (I have simplified it for ease of reading, sorry for the lack of functions): #include #include #include #include #include #include #include #include…
adrem7
  • 388
  • 1
  • 3
  • 14
0
votes
1 answer

Floating Point Precision using REST

I'm pulling timeseries data in from a MS-SQL database using REST. I've found that the floating point precision goes down from a value like 0.00166667 to 0.002 when I'm using REST to retrieve data, but using the DB designer's own tools, the…
qmoog
  • 205
  • 2
  • 13
0
votes
0 answers

Trouble with identical and all.equal when comparing two floating point elements in two vectors

This is a follow up to a question I posted yesterday. I can't seem to get things right about floating point comparison in R. Yesterday I was using >= to compare two floating point values but that dint seem to get the right results. Today, I tried…
Brian
  • 1,729
  • 2
  • 14
  • 17
0
votes
3 answers

Explain 1 - 3*(4/3 - 1) = 2.2204e-16 in Matlab

I am trying to understand the double precision numbers in Matlab. Why is this 1 - 3*(4/3 - 1) not equal to zero?
hhh
  • 50,788
  • 62
  • 179
  • 282
0
votes
1 answer

When casting float to int, why do numbers > -1 and < -0.5 go to -1, and numbers >= -0.5 and < 0 go to 0?

I always thought when floats get truncated to ints, it always becomes the highest number smaller than the float. However, when the float is between -0.5 and 0, it gets converted to 0 instead of -1! Why is this? Is this language-specific? I am seeing…
Popcorn
  • 5,188
  • 12
  • 54
  • 87
0
votes
1 answer

Integer multiplied by float and then divided by same float - a clean way to force value back to original integer

Alright so we have the classic 1 * 1/3 * 3 = 0.9999999. I understand why this is the case. In my use case, I am trying to result in an integer. I am happy to take the floor of my value, except when infinite precision math dictates that we should…
user321605
  • 846
  • 2
  • 7
  • 20
0
votes
2 answers

PHP function for float or double comarision (bccomp and gmp_cmp)

PHP Suggested to use these function for floating point number comparison [here] bccomp — Compare two arbitrary precision numbers int bccomp ( string $left_operand , string $right_operand [, int $scale ] ) gmp_cmp — Compare numbers int gmp_cmp (…
user1646111
0
votes
4 answers

Dividing integers in C rounds the value down / gives zero as a result

I'm trying to do some arithmetic on integers. The problem is when I'm trying to do division to get a double as a result, the result is always 0.00000000000000000000, even though this is obviously not true for something like ((7 * 207) / 6790). I…
user2052561
  • 1,339
  • 2
  • 15
  • 18
0
votes
1 answer

Getting more accurate float calculations

I'm building a game using the canvas element powered by JavaScript. Part of my player class includes an update() method which is called once per tick. In this method, I'm doing some math to update the players velocity based on keyboard input and…
James Dawson
  • 5,309
  • 20
  • 72
  • 126
0
votes
3 answers

Float precision with specific numbers

The following value gives me wrong precision. It is observed with only specific numbers. It might be a floating representation problem, but wanted to know the specific reason. String m = "154572.49"; //"154,572.49"; Float f =…
RP-
  • 5,827
  • 2
  • 27
  • 46
0
votes
2 answers

Looking for datatypes of similar precision in multiple programming languages e.g. C/C++, D, Go

I am trying to implement a program with floating point numbers, using two or more programming languages. The program does say 50k iterations to finally bring the error to very small value. To ensure that my results are comparable, I wanted to make…
user984260
  • 3,401
  • 5
  • 25
  • 38
0
votes
2 answers

MIL-STD-1750A to Decimal Conversion Examples

I am looking at some examples in the 1750A format webpage and some of the examples do not really make sense. I have included the 1750A format specification at the bottom of this post in case anyone isn't familiar with it. Take this example from…
halexh
  • 3,021
  • 3
  • 19
  • 19
1 2 3
…
90 91