Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
-1
votes
1 answer
What are the uses/applications of Single precision Floating point numbers?
I'm doing a project on single precision floating numbers. I was wondering in what fields or areas are these concepts used? Thanks in advance.
Manoj
- 23
- 6
-1
votes
1 answer
Rounding a floating number
I am trying to create a class dedicated to the floating point precision management.
Actually, it takes your double and the number of decimal it has to be rounded, then everytime you will modify it with set(double), the number will be automatically…
Antoine C.
- 3,730
- 5
- 32
- 56
-1
votes
1 answer
javascript float calculation
I have problem with calcualting float values via javascript.
first my code
var d_val = 0.00;
$.each($('.prices a b'), function(index,obj){
d_val = parseFloat( $(obj).text().replace( '€', '' ) );
group_price += d_val;
});
console.log(…
AppGeer
- 725
- 1
- 11
- 27
-1
votes
2 answers
storing values into double vector with same precision C++
Here is my problem. I have a vector suct that vector and after some operations I'll store some double values into this vector.However,I want they all have same decimal point digit. I mean if first value to be stored is 4.143 and the second one is…
caesar
- 2,865
- 11
- 29
- 36
-1
votes
3 answers
In Javascript: 1.1 - 1 = 0.10000000000000009
What kind of mistakes?
How do I fix this?
1.1 - 1 = 0.10000000000000009
1.0000001+1 = 2.0000001000000003
lukaszpolowczyk
- 605
- 1
- 7
- 27
-1
votes
4 answers
Python math accuracy
I have try this simple math operation in python
>>> math.floor(8.2-0.21)
7.0
>>> math.floor(8.21-0.2)
8.0
>>> math.floor(8.2-0.2)
7.0
The third one should return 8 but it is return 7?!
UPDATE
i have try it in PHP, ruby and JAVA and i have got the…
shox
- 1,150
- 5
- 18
- 32
-1
votes
1 answer
Concatenating matrix in MATLAB results in loss of precision
I have a column matrix say temp1 with floating point numbers in the following format(displayed this way using format long g):
1334320224.86767
1334320225.03415
1334320225.20064
and another nx3 matrix (temp2) with values like so:
25.59989 -17.82167…
user1984537
- 27
- 1
-1
votes
1 answer
C++ log precision error for double vector array?
I have double vector array ( vector< double > ) and its max value ( 1 ) and min value ( 0 ).
When I did below sub operation, log gives (1.#INF00000000000000000) for max value's index that ( 1 - 1.000000000000001 ) ( double precison ).
How can I…
user2227305
- 11
- 3
-1
votes
3 answers
Discarding 4 precision points?
I have a floating point number, but I only want the number till 2 points of precision. How do I get this in C++?
float foo(float num) { // num=1234.567891
// code
return num2; // returns 1234.560000
}
ofey
- 423
- 1
- 7
- 17
-1
votes
4 answers
Assigning 2^31 to a float or double
I have to multiply an integer value by 2^31. I have googled it, and it looks like doubles have a range among 2.23e-308 <= |X| <= 1.79e308 when using 64 bits, and a float among 1.18e-38 <= |X| <= 3.40e38.
That's a lot more than what I need. But it…
Roman Rdgz
- 12,836
- 41
- 131
- 207
-1
votes
1 answer
C++ Precision digits for float vs double
So I understand the difference for floats vs doubles in C++.
A double is simply a double float point whereas a float a single
floating point (double the size of precision).
My question is, how come floats are represented as…
Setheron
- 3,520
- 3
- 34
- 52
-1
votes
1 answer
How to store exponential values (1.3e12) and normal values(1.3) in the same matrix in matlab?
I have a file consisting of values ranging from 0.1 to 1.3e12. I have been trying it to store in the same array but its not working. Can anybody help?
SooryaS
- 42
- 4
-2
votes
1 answer
controlling overflow and loss in precision while multiplying doubles
ques:
I have a large number of floating point numbers (~10,000 numbers) , each having 6 digits after decimal. Now, the multiplication of all these numbers would yield about 60,000 digits. But the double range is for 15 digits only. The output…
bnks452
- 1
- 1
-2
votes
2 answers
Evaluate a math expression inside a String into a Float?
I need to evaluate a mathematical expression in Java.
The problem is, the expression is a String object.
Is there a way to take the string "( (21 + 3) / 4 )" and evaluate it so that it gives 6 as a result?
This is the code so far.
import…
vieuxdats
- 19
- 5
-2
votes
2 answers
I need help understanding ruby's floating point precision
I came across this the other day and sure its not causing me any trouble but i'm just curious as to why it happens?
1.9.2p320 :001 > 0.39-0.09
=> 0.30000000000000004
saGii
- 447
- 5
- 14