Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
3
votes
1 answer
How do I access rounding modes defined in IEEE 754 in python?
I have the need to compute exactly in single precision floating points in Python.
The options I've tried are decimal.Decimal and numpy.float32. However Decimal is not based on IEEE 754, and float32 cannot allow the use of rounding modes. It is…
koo
- 2,888
- 1
- 23
- 29
3
votes
3 answers
How to test whether a float is an integer "within machine precision"?
The brief version of my question is:
What's considered "best practice" for deciding when a floating point number x and Math.round(x) may be considered equal, allowing for loss of precision from floating-point operations?
The long-winded version…
kjo
- 33,683
- 52
- 148
- 265
3
votes
2 answers
How to get precise low p-value in R (from F test)
I'm computing a p-value from an F-test in R, and it seems to have trouble displaying anything lower than 1e-16. E.g., for F values from 80 to 90 with degrees of freedom 1,200:
> 1-pf(80:90,1,200)
[1] 2.220446e-16 2.220446e-16 1.110223e-16…
Stephen Turner
- 2,574
- 8
- 31
- 44
3
votes
3 answers
Switching between float and double precision at compile time
Where should I look at if I want to switch between float and double precision at compile time. Its like, if user wants everything in float instead of double precision how I can maintain this flexibility? In other words, how should I define a…
Shibli
- 5,879
- 13
- 62
- 126
3
votes
1 answer
What would the smallest positive value of an IEEE 24-bit float be?
The smallest positive value of an IEEE-754 32-bit float is 2^−149 and that of a 16-bit float is 2^-14. If the IEEE-754 standardized a 24-bit float, what would its smallest positive value be?
Excuse me if this is explicitly documented somewhere, but…
Ky -
- 30,724
- 51
- 192
- 308
3
votes
3 answers
Floating point operations ambiguity
Possible Duplicate:
Why is floating point arithmetic in C# imprecise?
Why is there a bias in floating point ops? Any specific reason?
Output:
160
139
static void Main()
{
float x = (float) 1.6;
int y = (int)(x *…
Amber
- 290
- 1
- 3
- 10
3
votes
1 answer
OpenCL speed and float point precision
I have just started working with OpenCL. However, I have found some weird behavior of OpenCl, which i can't understand. The source i built and tested, was http://www.codeproject.com/Articles/110685/Part-1-OpenCL-Portable-Parallelism . I have a ATI…
Tapped
- 153
- 2
- 9
3
votes
1 answer
parseDecimal in JavaScript?
I have some JS calculations going on. Since floating point arithmetic often uses close approximations to numbers instead of exact values, if you round these floating point number to fixed precision, you often get slight differences. When you are…
jeffery_the_wind
- 17,048
- 34
- 98
- 160
3
votes
1 answer
Inconsistent summation in C
When I run the following program:
#include
#include
int main()
{
double sum, increase;
long amount, j;
printf("sum = ");
scanf("%lf", &sum);
printf("increase = ");
scanf("%lf", &increase);
…
Benjamin
- 29
- 2
3
votes
2 answers
Coping with big ints in 32 bit PHP
I have a class for computing the Luhn checksum for a number. It takes integer as an input and returns true or false to indicate validity or otherwise, or it throws an exception if an inappropriate data type is given as input.
The code is as…
GordonM
- 31,179
- 15
- 87
- 129
3
votes
2 answers
Multiple Precision for Visual Studio 2010
Can anyone suggest a multiple precision library that comes with pre-built windows binaries. I need to use it use with an existing Visual Studio 2010 project (or where I can get pre-built binaries for win7 64bit for GMP).
I have tried and failed to…
user1392800
- 31
- 2
3
votes
2 answers
Do doubles suffer from overflow?
Is it possible to have an overflow (wrap around) with a double or a float? What will happen if the maximum (or minimum) value is reached on x86 or x64 hardware?
unixman83
- 9,421
- 10
- 68
- 102
3
votes
1 answer
Calling DLL From VB6 and C# Give Slightly Different Results in Double Precision
I have a proprietary library in a DLL (I don't have the code) that has been used for years from within VB6. I'm trying to upgrade the VB6 code to C#, and hope to make the C# code exactly replicate the VB6 behavior. I'm having trouble making the…
user12861
- 2,358
- 4
- 23
- 41
2
votes
2 answers
Why do my float values lose precision and drop decimals? Example code below
I have never used floats really that much before and the current project I am working on requires them. I'm getting weird issues that I learned about years ago but have forgotten why this happens.
My results after multiplying or adding floats aren't…
CREW
- 926
- 3
- 17
- 32
