Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
3
votes
1 answer
Widening precision with cast results in how much precision?
I know that widening conversions are safe in that they result in no loss of data, but is there a real gain in precision or is it a longer representation with the same number of signifigant figures?
For example,
#include
#include…
Cecilia
- 4,512
- 3
- 32
- 75
3
votes
3 answers
Can we assume any pair of following floating point arithmetic statements always produce identical result?
Given following floating points variables with arbitrary value, in c/c++.
float a, b, c, d;
In following statements, can we assume any pair of them would always generate identical result?
float result_1 = a + b + c + d - c;
float result_2= a + b +…
mr49
- 1,053
- 1
- 8
- 26
3
votes
4 answers
Working with massive numbers and minute numbers simultaneously
I have an mxn matrix A, with very small or very large elements. For example:
A = -1e4*randn(10,20);
I would like to create a new matrix C of the same size, as follows:
First, define a matrix B whose elements are the exponential of the elements of…
f10w
- 1,524
- 4
- 24
- 39
3
votes
1 answer
XML After Effect File (*.aepx) -> Understand binary number format to edit xml file
I'm trying to understand number format for aepx file for After Effect CS6 and CC
Coordinates are coded in cdat hexadecimal data. Coordinates is two number. I have made a list of number with the encoded hexadecimal value to help to understand format…
VanVan
- 31
- 4
3
votes
2 answers
C++ precision of numbers and truncation with fstream
I have a file.txt with hundreds of numbers.
They have many digits (max 20) after the point and I need to get them all without truncation, otherwise they introduce errors in the following computations. I made these numbers with matlab so it has a…
user3290180
- 4,260
- 9
- 42
- 77
3
votes
2 answers
Decimal accuracy of binary floating point numbers
I've found this problem in many interview exams, but don't see how to work out the proper solution myself. The problem is:
How many digits of accuracy can be represented by a floating point number represented by two 16-bit words?
The solution is…
jone kim
- 51
- 1
- 6
3
votes
1 answer
Decimal(str(my_float)) seems to be better than Decimal(my_float), what's going on?
I'm reading a JSON data file that might give me a float value of, say, 1.1. When I make a Decimal of that value, I get a crazy long number, because of the imprecision of binary representations of floats.
I understand binary representation and I'm…
Daniel Baird
- 2,239
- 1
- 18
- 24
3
votes
2 answers
Whats the difference in these two ways to convert milliseconds to seconds?
First way:
long mySeconds = milliseconds/ 1000;
Second way:
double mySeconds = milliseconds * 1e-3d;
This calculation is finally used to determine index of an array, like this:
int index = (int) ((someDoubleSeconds + mySeconds)/…
user1071840
- 3,522
- 9
- 48
- 74
3
votes
2 answers
Incorrect floating point rounding
On gcc 4.7.3, my fegetround() function returns FE_TONEAREST. According to the c++ reference, this means rounding away from zero. Essentially, it means saving the last bit that was shifted out when adjusting the precision of the mantissa after…
Suedocode
- 2,504
- 3
- 23
- 41
3
votes
2 answers
Floating point precision variation in Python 2.7.5
If I run the following code in Python 2.7.5 console:
>>> import math
>>> math.radians(0.000001)
I get
1.7453292519943295e-08
However, if I put the same code in a file:
$ cat floatingtest.py
import math
print(math.radians(0.000001))
And run it, I…
Kedar
- 1,648
- 10
- 20
3
votes
1 answer
Why is the behavior of Double.toString not the same as MathContext.DECIMAL64 when constructing a BigDecimal in Java?
To me, it seems like Double should be following the same rules as the IEEE standard used in MathContext.DECIMAL64, however, in this case, I get different behavior:
import java.math.BigDecimal;
import java.math.MathContext;
public class…
dave
- 1,607
- 3
- 16
- 20
3
votes
2 answers
How to calculate floating-point precision after round-off errors in +, -, *, and /?
For the purpose of verification, I would like to be able calculate a reasonably tight upper bound on the accumulated error due to rounding to representable values during some specific arithmetic computation.
Assume that we have a function foo() that…
Kristian Spangsege
- 2,903
- 1
- 20
- 43
3
votes
3 answers
temperature conversion for python
I'm taking a free online Python tutorial, which wants me to:
Create a temperature converter which will convert Fahrenheit values to
Celsius and vice-versa using the following two formulas which relate
the temperature f in Fahrenheit to the…
iaianmcd
- 117
- 1
- 4
- 13
3
votes
4 answers
Why does this simple double assertion fail in C#
The following test will fail in C#
Assert.AreEqual(10.0d, 16.1d - 6.1d);
The problem appears to be a floating point error.
16.1d - 6.1d == 10.000000000000002
This is causing me headaches in writing unit tests for code that uses double. Is…
Reactgular
- 52,335
- 19
- 158
- 208
3
votes
1 answer
nVidia GPUs for Research Purposes: Float Precision
I'm doing my PhD research in A.I. and I've gotten to the part where I have to start using CUDA libraries for my testing platform.
I've played with CUDA before, and I have a basic understanding of how GPGPU works, etc, but I am troubled by the float…
Ælex
- 14,432
- 20
- 88
- 129