Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
4
votes
1 answer
How are the TI operating systems coded not to have floating point error?
I ran a test on my graphing calculator to check for floating point error, and after forty eight hours of complete and utter randomness, the calculator had not lost a single digit of precision.
How does TI pull this off?
user1131435
4
votes
4 answers
Precision issues with Visual Studio 2010
I have an application written in Microsoft Visual C++ 6.0. Now I have rewritten the application in Visual Studio 2010 using C#, but the results are not matching because of precision problems. One of such precision issues is the following one.
float…
Mahesh
- 169
- 9
4
votes
4 answers
Ruby - Multiplication issue
My output is like this -
ruby-1.9.2-p290 :011 > 2.32 * 3
=> 6.959999999999999
And I remember sometime back on another machine I had got it like.. 2.32 * 3 = 6
What is my mistake?
Thanks a ton for reading this. :)
Anuprit
- 95
- 1
- 4
4
votes
1 answer
Output precision is higher than double precision
I am printing some data from a C++ program to be processed/visualized by ParaView, but I am having a problem with floating point numbers. Paraview supports both Float32 and Float64 data types. Float64 is equivalent to double with the typical limits …
iluvatar
- 872
- 10
- 21
4
votes
2 answers
Extended precision floating point dangers in C#
I am writing a library for multiprecision arithmetic based on a paper I am reading. It is very important that I am able to guarantee the properties of floating point numbers I use. In particular, that they adhere to the IEEE 754 standard for double…
Void Star
- 2,401
- 4
- 32
- 57
4
votes
3 answers
Ensure float to be smaller than exact value
I want to calculate a sum of the following form in C++
float result = float(x1)/y1+float(x2)/y2+....+float(xn)/yn
xi,yi are all integers. The result will be an approximation of the actual value. It is crucial that this approximation is smaller or…
Ricardo
- 245
- 1
- 9
4
votes
2 answers
Converting SQL Geography Lat/Long to VARCHAR without loss of precision
I have a geography column called Location.
I need to SELECT the Location.Lat and Location.Long as a single VARCHAR result.
Using the following I am losing precision:
Query:
SELECT CONVERT(VARCHAR(MAX), Location.Lat) + ' ' + CONVERT(VARCHAR(MAX),…
KrustyGString
- 903
- 2
- 13
- 32
4
votes
1 answer
How to reproduce floating point cos(x)!=cos(x)
How to reproduce this behavior?
https://isocpp.org/wiki/faq/newbie#floating-point-arith2
To be precise, in the following code, parameters x and y are equal; they can be equal to 1.0 or any other value.
void foo(double x, double y)
{
double cos_x =…
userbb
- 2,148
- 5
- 30
- 53
4
votes
2 answers
How to raise the precision of pow in C++ for large numbers (10^19)?
I am looking to calculate 9^19. my code is:
cout.setf(ios::fixed, ios::floatfield);
cout.setf(ios::showpoint);
cout<
Assem
- 11,574
- 5
- 59
- 97
4
votes
1 answer
Is there any way to influence the code the F# compiler generates for structural equality?
Here is a specific example.
I have a record type with floating point fields and I would like to be able to compare instances of this type for equality but of course the default compiler-generated implementation of structural equality and comparison…
dmg
- 608
- 8
- 15
4
votes
2 answers
Standard guarantees for using floating point arithmetic to represent integer operations
I am working on some code to be run on a very heterogeneous cluster. The program performs interval arithmetic using 3, 4, or 5 32 bit words (unsigned ints) to represent high precision boundaries for the intervals. It seems to me that representing…
Jack
- 169
- 7
4
votes
1 answer
Last decimal digit precision changes in different call of same generator function [python]
I created this generator function:
def myRange(start,stop,step):
r = start
while r < stop:
yield r
r += step
and I use it in two different ways. 1st:
for x in myRange(0,1,0.1):
print…
Alvin
- 383
- 5
- 16
4
votes
3 answers
Machine epsilon vs least positive number
What is the difference between machine epsilon and least positive number in floating point representation?
If I try to show the floating point number on a number line .Is the gap between exact 0 and the first positive (number which floating point…
monkey
- 561
- 1
- 7
- 16
4
votes
1 answer
In SQL Server, why are all decimals between .00 and .99 represented accurately in float fields?
For decimal numbers of the form [x.00, x.99], where x is any arbitrary integer (you can probably guess part of the context here), floating-point datatypes are generally considered to be fairly inaccurate and sometimes dangerous. Instead, in…
Panzercrisis
- 4,590
- 6
- 46
- 85
4
votes
1 answer
C++ Builder is StrToFloat() the same as .ToDouble()
I'm working in a XE6 project, but this may apply to other version of builder as well.
I'm looking at a function name, which I think may be misleading. I'm curious if StrToFloat() returns a float or if it returns a double. I found an alternative,…
James Oravec
- 19,579
- 27
- 94
- 160