Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
12
votes
3 answers
C++ sqrt function precision for full squares
Let, x is an integer and y = x * x.
Then is it guaranteed that sqrt(y) == x?
For example, can I be sure that sqrt(25) or sqrt(25.0) will return 5.0, not 5.0000000003 or 4.999999998 ?
Rafi Kamal
- 4,522
- 8
- 36
- 50
10
votes
6 answers
How to convert floating value to integer with exact precision like 123.3443 to 1233443?
Sample code:
int main() {
float f = 123.542;
int i = (int)f;
printf("%d\n",i);
}
sur
- 345
- 1
- 3
- 13
9
votes
5 answers
Why float variable saves value by cutting digits after point in a weird way?
I have this simple code line:
float val = 123456.123456;
when i print this val or look in scope, it stores value 123456.13
Ok, it's fine, it can't store all those digits after point just in 4 bytes, but why does it make 13 after the point?…
Kosmo零
- 4,001
- 9
- 45
- 88
9
votes
2 answers
Is this a valid float comparison that accounts for a set number of decimal places?
I'm writing an extension method to compare two floats using a set number of decimal points (significant figures) to determine if they are equal instead of a tolerance or percentage difference. Looking through the other questions regarding float…
Kim
- 1,068
- 13
- 25
9
votes
7 answers
Solving floating-point rounding issues C++
I develop a scientific application (simulation of chromosomes moving in a cell nucleus). The chromosomes are divided in small fragments that rotate around a random axis using 4x4 rotation matrices.
The problem is that the simulation performs…
user703016
- 37,307
- 8
- 87
- 112
9
votes
5 answers
Can cout alter variables somehow?
So I have a function that looks something like this:
float function(){
float x = SomeValue;
return x / SomeOtherValue;
}
At some point, this function overflows and returns a really large negative value. To try and track down exactly where…
Jason Baker
- 192,085
- 135
- 376
- 510
8
votes
1 answer
Lodash rounding precision
I'm trying to display a number as a percent by using _.round and then by multiplying the number by 100. For some reason, when I multiply the rounded number, the precision gets messed up. Here's what it looks like:
var num = 0.056789,
…
Tim Perkins
- 2,339
- 3
- 20
- 13
8
votes
3 answers
In Java, when is a float equal to zero?
For what values of x does the test (x == 0) return true? Is there some kind of margin or does the test return true if and only if the value of x = 0?
Wilco
- 928
- 3
- 9
- 20
8
votes
1 answer
PostgreSQL: what is the difference between float(1) and float(24)?
I am having a hard time understanding the precision parameter p for float(p) in PostgreSQL. For example, float(1) and float(24) seem to be exactly the same to me.
Can anyone provide me with some examples of their differences, please?
Xin
- 737
- 3
- 10
- 27
8
votes
2 answers
SELECT a FLOAT with given precision
I want to store my values as FLOATS, but when retrieving them the user can specify (and change at runtime) how many digits he wants to see after the decimal point.
Can I pass that in my SELECT somehow, or do I have to do it in my application?
Mawg says reinstate Monica
- 38,334
- 103
- 306
- 551
7
votes
6 answers
How to count from 0.0001 to 1 in ruby?
I want to count from 0.0001 to 1 with 0.0001 steps in ruby.
I wrote this code but it enters to an infinite loop. Somewhy the interpreter does a wrong summation.
x = 0.0001
while x != 1.0
puts x
x = x + 0.0001
end
Here is the first 10 value it…
Hiddin
- 75
- 4
7
votes
1 answer
Exactly storing large integers
In R software
a <- 123456789123456789123456789
sprintf("%27f",a)
#[1] "123456789123456791337762816.000000"
I got the wrong answer. I want exact a value.
Why is the system showing the wrong value of a?
aarthi
- 85
- 5
7
votes
3 answers
How to round a floating point number upto 3 digits after decimal point in bash
I am a new bash learner. I want to print the result of an expression given as input having 3 digits after decimal point with rounding if needed.
I can use the following code, but it does not round. Say if I give 5+50*3/20 + (19*2)/7 as input for the…
Enamul Hassan
- 5,266
- 23
- 39
- 56
7
votes
2 answers
Converting Int to Float loses precision for large numbers in Swift
XCode 6.3.1 Swift 1.2
let value: Int = 220904525
let intmax = Int.max
let float = Float(value) // Here is an error probably
let intFromFloat = Int(float)
let double = Double(value)
println("intmax=\(intmax) value=\(value) float=\(float)…
kostyl
- 339
- 3
- 15
7
votes
4 answers
Javascript Math.floor issue between specific range of numbers
I'm facing an issue with Math.floor function of javascript for the below scenario:
1) from the value betwwen 8192 and 10484,
if I type 8192.8 -> The Math.floor converts it into 8192.79
if I type 8192.88 -> The Math.floor converts it into…
ron_meh
- 93
- 1
- 6