Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
57
votes
4 answers

Inaccuracy of decimal in .NET

Yesterday during debugging something strange happened to me and I can't really explain it: So maybe I am not seeing the obvious here or I misunderstood something about decimals in .NET but shouldn't the results be the same?
10rotator01
  • 635
  • 6
  • 15
57
votes
1 answer

What is the standard solution in JavaScript for handling big numbers (BigNum)?

Is there a bignum built into JavaScript or browsers? The alternate is loading an external library like but that seems slow and may trigger a security warning. I've considered…
David Cary
  • 5,250
  • 6
  • 53
  • 66
53
votes
17 answers

How to do an integer log2() in C++?

In the C++ standard libraries I found only a floating point log method. Now I use log to find the level of an index in a binary tree ( floor(2log(index)) ). Code (C++): int targetlevel = int(log(index)/log(2)); I am afraid that for some of the…
Peter Smit
  • 27,696
  • 33
  • 111
  • 170
50
votes
11 answers

How to get bc to handle numbers in scientific (aka exponential) notation?

bc doesn't like numbers expressed in scientific notation (aka exponential notation). $ echo "3.1e1*2" | bc -l (standard_in) 1: parse error but I need to use it to handle a few records that are expressed in this notation. Is there a way to get bc to…
Ferdinando Randisi
  • 4,068
  • 6
  • 32
  • 43
48
votes
2 answers

Why float.Epsilon and not zero?

In the following code, why is there a comparison against float.Epsilon and not 0? // Coroutine to move elements protected IEnumerator SmoothMovement (Vector3 end) { // Distance computation float sqrRemainingDistance = (transform.position -…
Olivier Pons
  • 15,363
  • 26
  • 117
  • 213
42
votes
10 answers

Getting the decimal part of a double in Swift

I'm trying to separate the decimal and integer parts of a double in swift. I've tried a number of approaches but they all run into the same issue... let x:Double = 1234.5678 let n1:Double = x % 1.0 // n1 = 0.567800000000034 let n2:Double =…
user3925713
  • 509
  • 1
  • 4
  • 10
40
votes
8 answers

How to correctly and standardly compare floats?

Every time I start a new project and when I need to compare some float or double variables I write the code like this one: if (fabs(prev.min[i] - cur->min[i]) < 0.000001 && fabs(prev.max[i] - cur->max[i]) < 0.000001) { continue; } Then…
Dmitriy
  • 5,357
  • 8
  • 45
  • 57
38
votes
1 answer

When to use `std::hypot(x,y)` over `std::sqrt(x*x + y*y)`

The documentation of std::hypot says that: Computes the square root of the sum of the squares of x and y, without undue overflow or underflow at intermediate stages of the computation. I struggle to conceive a test case where std::hypot should be…
Emily L.
  • 5,673
  • 2
  • 40
  • 60
38
votes
9 answers

What is a simple example of floating point/rounding error?

I've heard of "error" when using floating point variables. Now I'm trying to solve this puzzle and I think I'm getting some rounding/floating point error. So I'm finally going to figure out the basics of floating point error. What is a simple…
MrDatabase
  • 43,245
  • 41
  • 111
  • 153
34
votes
3 answers

Is there any accuracy gain when casting to double and back when doing float division?

What is the difference between two following? float f1 = some_number; float f2 = some_near_zero_number; float result; result = f1 / f2; and: float f1 = some_number; float f2 = some_near_zero_number; float result; result = (double)f1 /…
Piotr Lopusiewicz
  • 2,514
  • 2
  • 27
  • 38
33
votes
4 answers

Is integer division always equal to the floor of regular division?

For large quotients, integer division (//) doesn't seem to be necessarily equal to the floor of regular division (math.floor(a/b)). According to Python docs (https://docs.python.org/3/reference/expressions.html - 6.7), floor division of integers…
Aditya Chanana
  • 441
  • 4
  • 6
33
votes
2 answers

mean from pandas and numpy differ

I have a MEMS IMU on which I've been collecting data and I'm using pandas to get some statistical data from it. There are 6 32-bit floats collected each cycle. Data rates are fixed for a given collection run. The data rates vary between 100Hz and…
Rob
  • 335
  • 3
  • 5
33
votes
12 answers

How to avoid floating point precision errors with floats or doubles in Java?

I have a very annoying problem with long sums of floats or doubles in Java. Basically the idea is that if I execute: for ( float value = 0.0f; value < 1.0f; value += 0.1f ) System.out.println( value ); What I get…
tunnuz
  • 23,338
  • 31
  • 90
  • 128
32
votes
5 answers

How to Java String.format with a variable precision?

I'd like to allow the user to vary the precision of a String generated from a double. Right now, I'm trying something like String foo = String.format("%.*f\n", precision, my_double); However, I receive a java.util.UnknownFormatConversionException.…
Willi Ballenthin
  • 6,444
  • 6
  • 38
  • 52
31
votes
1 answer

Is "banker's rounding" really more numerically stable?

By banker's rounding I mean "round to nearest, ties to even" as recommended by IEEE 754: Rounds to the nearest value; if the number falls midway it is rounded to the nearest value with an even (zero) least significant bit. This is the default for…
Luis Mendo
  • 110,752
  • 13
  • 76
  • 147
1
2
3
…
90 91