Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
-2
votes
3 answers
Representation of Long Double value
I want to know how a value is represnted in long double.
Is it as L or LF . Im asking this since in some places i have read it is L and in some LF.
In one of the exam answers and in some websites like indiabix it was L and in some stack overflow…
mahashwetha
- 71
- 7
-2
votes
2 answers
how to eliminate any number after the floating point?
Is there any function returns only the real number without the floating point? For an example,
func(1.xxx)->1
func(52.xx)->52
func(0.xx)->0
Is there any function does so?
Amrmsmb
- 1
- 27
- 104
- 226
-2
votes
1 answer
C++ float rounding (error?)
Why when I use multiplication
float a= 1.0500000f;
float b= a*100.0f;
why b is 104.99999 but not 105.0 ?
and when I
int f= (int)b;
f is 104
mrgloom
- 20,061
- 36
- 171
- 301
-2
votes
1 answer
Inconsistent value in floating point operations JAVA
I don't know whether this question is asked before. I couldn't find it in searches.
System.out.println(1.00 - 9*.10);
the above statement prints out 0.09999999999999998 instead of simply 0.01
I know this is a well known problem in JAVA but i want…
Manu Viswam
- 1,606
- 2
- 18
- 33
-2
votes
2 answers
calculate decimal fraction and show digits after decimal in a precise manner
Given a fraction (103993/33102), I need to find 50000 digits after the decimal point for this fraction.
Initially I used setprecision(k) in C++, but it gives only 17 digits after decimal point. I also tried
sprintf (str, "%.500000f", num)
but the…
user2119531
- 69
- 1
- 1
- 3
-3
votes
1 answer
IEEE-754 single precision floating point mips
How I write -2^-132 in hexadecimal format in IEEE-754 single precision?
anil cheetri
- 1
- 1
- 2
-3
votes
1 answer
weird result after calling a method that should return a floating number with exact number of decimals
I call the following method to convert the inputs as indicated in the body :
public static double convertFromHectareAreCentiareToHectare(int hectare, int are, int centiare){
double d;
DecimalFormat df = new…
mounaim
- 1,132
- 7
- 29
- 56
-3
votes
1 answer
Python: 2.2*3.0=6.6000000000000005
My question is more about
Why in C++,C# and VB
std::cout<<2.2f*3.0f<
Sam Aleksov
- 1,201
- 6
- 12
-5
votes
1 answer
Manually doing soft-float add
I need to write a program in C that takes 2 floating point numbers and produces their sum as the output only using int, pointers and malloc(). No other data type. (Single precision)
And I'm helpless.
user3504047
- 1
- 3
-5
votes
2 answers
How to get the answer to exact bits using double in android
I have made a Calculator app on android but whenever I do some multiplication of the kind:
23.3 x 3.3
The answer I get is something like:
76.988999999999
Now please tell me how to resolve this conflict with this app. The answer should be…
Himanshu Aggarwal
- 1,803
- 2
- 24
- 36
-9
votes
3 answers
How a Double.ToString() displays more chars that Double's precision?
?(1.0-0.9-0.1)
-0.000000000000000027755575615628914
?((double)1.0-(double)0.9-(double)0.1)
-0.000000000000000027755575615628914
?((double)1.0-(double)0.9-(double)0.1).GetType()
{Name = "Double" FullName =…
serhio
- 28,010
- 62
- 221
- 374