Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
7
votes
3 answers
How do C and C++ compilers implement the decision of equality for float numbers?
For example,
float a = 1.0;
float b = 1.2;
puts(a == b? "equal": "not equal");
Does compiler deal with it bitwisely or by some other methods?
(I know it's not a good choice to decide the equality of floating-point numbers by "==", I just want to…
Wizmann
- 839
- 1
- 9
- 14
7
votes
3 answers
Engineering notation with Haskell
Is there a existing Haskell function which provide an engineering notation formatting (as String)?
If not, I read that printf can be extended by adding an instance to PrintfArg. Do you believe this is a good solution ?
By engineering notation, I…
JeanJouX
- 2,555
- 1
- 25
- 37
7
votes
1 answer
Setting precision in Decimal class Python
I just learned the Decimal class in Python, and I have some issues in modifying the precision of the decimal numbers. Code:
from decimal import *
def main() :
getcontext().prec = 50
print Decimal(748327402479023).sqrt()
if __name__ ==…
Rontogiannis Aristofanis
- 8,883
- 8
- 41
- 58
7
votes
1 answer
.format() returns ValueError when using {0:g} to remove trailing zeros
I'm trying to generate a string that involves an occasional float with trailing zeros. This is a MWE of the text string and my attempt at removing them with {0:g}:
xn, cod = 'r', 'abc'
ccl = [546.3500, 6785.35416]
ect = [12.350, 13.643241]
text =…
Gabriel
- 40,504
- 73
- 230
- 404
7
votes
3 answers
How does floating point error propagate when doing mathematical operations in C++?
Let's say that we have declared the following variables
float a = 1.2291;
float b = 3.99;
float variables have precision 6, which (if I understand correctly) means that the difference between the number that the computer actually stores and the…
ksm001
- 3,772
- 10
- 36
- 57
7
votes
4 answers
What precision are floating-point arithmetic operations done in?
Consider two very simple multiplications below:
double result1;
long double result2;
float var1=3.1;
float var2=6.789;
double var3=87.45;
double var4=234.987;
result1=var1*var2;
result2=var3*var4;
Are multiplications by default done in a higher…
Pooria
- 2,017
- 1
- 19
- 37
7
votes
1 answer
Power of number close to 1
I'm guessing there is some standard trick that I wasn't able to find: Anyway I want to compute a large power of a number very close to 1(think 1-p where p<1e-17) in a numerically stable fashion. 1-p is truncated to 1 on my system.
Using the taylor…
Tobias Madsen
- 857
- 8
- 10
7
votes
1 answer
Less compiler uses different floating point precision on Linux vs OSX
Whenever I compile less files to css on my computer (Ubuntu 13.10, AMD64) I get a very different floating point result than when my colleague compiles on his machine (Mac OSX 10.6). Which is all well and good. Floating point math etc, etc. The…
aychedee
- 24,871
- 8
- 79
- 83
7
votes
2 answers
Does C round floating-point constants
A question about floating-point precision in Go made me wonder how C handles the problem.
With the following code in C:
float a = 0.1;
Will a have the closest IEEE 754 binary representation of:
00111101110011001100110011001101 (Decimal: …
ANisus
- 74,460
- 29
- 162
- 158
7
votes
1 answer
C# float precision
Can anyone please explain to me what's happening here:
using System;
using System.Text;
namespace ConsoleApplication1 {
class Program {
static void Main(string[] args) {
object o = 1000000.123f;
float f=…
Stephen Emm
- 83
- 1
- 1
- 5
7
votes
1 answer
Converting object to float loses too much precision - pandas
I'm trying to plot a DataFrame using pandas but it's not working (see this similar thread for details). I think part of the problem might be that my DataFrame seems to be made of objects:
>>> df.dtypes
Field object
Moment …
thosphor
- 2,493
- 7
- 26
- 42
7
votes
1 answer
Generate large number of unique random float32 numbers
I need to generate a binary file containing only unique random numbers, with single precision.
The purpose is then to calculate the entropy of this file and use it with other datasets entropy to calculate a ratio entropy_file/entropy_randUnique.…
SamGamgee
- 564
- 1
- 4
- 12
7
votes
3 answers
Java float is more precise than double?
Code:
class Main {
public static void main (String[] args) {
System.out.print("float: ");
System.out.println(1.35f-0.00026f);
System.out.print("double: ");
System.out.println(1.35-0.00026);
}
}
Output:
float:…
Mustafa
- 931
- 1
- 13
- 26
7
votes
4 answers
Euler #26, how to convert rational number to string with better precision?
I want to get 1/7 with better precision, but it got truncated. How can I get better precision when I convert a rational number?
>>> str(1.0/7)[:50]
'0.142857142857'
grokus
- 18,046
- 9
- 29
- 35
7
votes
4 answers
What is a good way to round double-precision values to a (somewhat) lower precision?
My problem is that I have to use a thrid-party function/algorithm which takes an array of double-precision values as input, but apparently can be sensitive to very small changes in the input data. However for my application I have to get identical…
MuldeR
- 577
- 1
- 4
- 17