Questions tagged [floating-point-precision]

Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.

551 questions
7
votes
3 answers

How do C and C++ compilers implement the decision of equality for float numbers?

For example, float a = 1.0; float b = 1.2; puts(a == b? "equal": "not equal"); Does compiler deal with it bitwisely or by some other methods? (I know it's not a good choice to decide the equality of floating-point numbers by "==", I just want to…
Wizmann
  • 839
  • 1
  • 9
  • 14
7
votes
3 answers

Engineering notation with Haskell

Is there a existing Haskell function which provide an engineering notation formatting (as String)? If not, I read that printf can be extended by adding an instance to PrintfArg. Do you believe this is a good solution ? By engineering notation, I…
7
votes
1 answer

Setting precision in Decimal class Python

I just learned the Decimal class in Python, and I have some issues in modifying the precision of the decimal numbers. Code: from decimal import * def main() : getcontext().prec = 50 print Decimal(748327402479023).sqrt() if __name__ ==…
7
votes
1 answer

.format() returns ValueError when using {0:g} to remove trailing zeros

I'm trying to generate a string that involves an occasional float with trailing zeros. This is a MWE of the text string and my attempt at removing them with {0:g}: xn, cod = 'r', 'abc' ccl = [546.3500, 6785.35416] ect = [12.350, 13.643241] text =…
Gabriel
  • 40,504
  • 73
  • 230
  • 404
7
votes
3 answers

How does floating point error propagate when doing mathematical operations in C++?

Let's say that we have declared the following variables float a = 1.2291; float b = 3.99; float variables have precision 6, which (if I understand correctly) means that the difference between the number that the computer actually stores and the…
ksm001
  • 3,772
  • 10
  • 36
  • 57
7
votes
4 answers

What precision are floating-point arithmetic operations done in?

Consider two very simple multiplications below: double result1; long double result2; float var1=3.1; float var2=6.789; double var3=87.45; double var4=234.987; result1=var1*var2; result2=var3*var4; Are multiplications by default done in a higher…
7
votes
1 answer

Power of number close to 1

I'm guessing there is some standard trick that I wasn't able to find: Anyway I want to compute a large power of a number very close to 1(think 1-p where p<1e-17) in a numerically stable fashion. 1-p is truncated to 1 on my system. Using the taylor…
7
votes
1 answer

Less compiler uses different floating point precision on Linux vs OSX

Whenever I compile less files to css on my computer (Ubuntu 13.10, AMD64) I get a very different floating point result than when my colleague compiles on his machine (Mac OSX 10.6). Which is all well and good. Floating point math etc, etc. The…
aychedee
  • 24,871
  • 8
  • 79
  • 83
7
votes
2 answers

Does C round floating-point constants

A question about floating-point precision in Go made me wonder how C handles the problem. With the following code in C: float a = 0.1; Will a have the closest IEEE 754 binary representation of: 00111101110011001100110011001101 (Decimal: …
ANisus
  • 74,460
  • 29
  • 162
  • 158
7
votes
1 answer

C# float precision

Can anyone please explain to me what's happening here: using System; using System.Text; namespace ConsoleApplication1 { class Program { static void Main(string[] args) { object o = 1000000.123f; float f=…
7
votes
1 answer

Converting object to float loses too much precision - pandas

I'm trying to plot a DataFrame using pandas but it's not working (see this similar thread for details). I think part of the problem might be that my DataFrame seems to be made of objects: >>> df.dtypes Field object Moment …
thosphor
  • 2,493
  • 7
  • 26
  • 42
7
votes
1 answer

Generate large number of unique random float32 numbers

I need to generate a binary file containing only unique random numbers, with single precision. The purpose is then to calculate the entropy of this file and use it with other datasets entropy to calculate a ratio entropy_file/entropy_randUnique.…
7
votes
3 answers

Java float is more precise than double?

Code: class Main { public static void main (String[] args) { System.out.print("float: "); System.out.println(1.35f-0.00026f); System.out.print("double: "); System.out.println(1.35-0.00026); } } Output: float:…
7
votes
4 answers

Euler #26, how to convert rational number to string with better precision?

I want to get 1/7 with better precision, but it got truncated. How can I get better precision when I convert a rational number? >>> str(1.0/7)[:50] '0.142857142857'
grokus
  • 18,046
  • 9
  • 29
  • 35
7
votes
4 answers

What is a good way to round double-precision values to a (somewhat) lower precision?

My problem is that I have to use a thrid-party function/algorithm which takes an array of double-precision values as input, but apparently can be sensitive to very small changes in the input data. However for my application I have to get identical…
MuldeR
  • 577
  • 1
  • 4
  • 17