Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
2
votes
1 answer
Precision loss with GMP
I have a program that reads numbers from a string to an mpz_t and then converts it to an mpf_t. Despite of being read correctly from the file, there is a precision loss when I convert them to mpf_t. The code is the following:
#include…
fc67
- 409
- 5
- 17
2
votes
2 answers
Strange output when using float instead of double
Strange output when I use float instead of double
#include
void main()
{
double p,p1,cost,cost1=30;
for (p = 0.1; p < 10;p=p+0.1)
{
cost = 30-6*p+p*p;
if (cost
Vivek Aditya
- 1,145
- 17
- 46
2
votes
1 answer
Converting raw binary to float printing 0 for small values
I am trying to read 2 32 bit values ( assuming they are ieee 745 single precision float values ) and comparing them to check if they are equal using following snippet,
#include
#include
void main(){
int a;
int b;
…
samkit
- 79
- 1
- 6
2
votes
4 answers
Can javascript be trusted when making calculations?
I am implementing an invoice system, where everything is dynamically added on the dom through javascript and I am making some calculations on the browser itself with javascript.
for eg I am calculating each invoice line with quantity and price of…
webmaster
- 1,960
- 24
- 29
2
votes
2 answers
Significant precision difference in floating point calculations in Java / C#
I know that similar questions have been asked before, but none of the answers solves my problem.
I have 2 functions:
Java
public static void main(String[] args) {
double h, z, lat0, n0, eSq;
z = 4488055.516;
lat0 =…
zoran
- 943
- 11
- 22
2
votes
2 answers
Rounding problems in JavaScript
I've got a problem with rounding in JavaScript. I'm using a function for rounding:
function roundup(rnum, rlength){
var newnumber = Math.round(rnum * Math.pow(10, rlength)) / Math.pow(10, rlength);
return newnumber;
}
var amount =…
Leon van der Veen
- 1,652
- 11
- 42
- 60
2
votes
2 answers
setprecision and floating point
Running the following code I expect to receive such an output:
Desired output:
Car Hours Charge
1 1.5 2.00
2 4.0 2.50
3 24.0 10.00
But the results comes out as follows:
Actual output:
Car Hours …
SRYZDN
- 307
- 5
- 13
2
votes
0 answers
Unexpected output in R with modulus operator
((10*(7655.7-7652.3))%/%(2))
[1] 16
((10*(655.7-652.3))%/%(2))
[1] 17
((10*(7655.7-7652.3))%/%(2))
[1] 16
((10*(8655.7-8652.3))%/%(2))
[1] 17
((10*(9655.7-9652.3))%/%(2))
[1] 17
((10*(7655.7-7652.3))%/%(2))
[1] 16
%/% operator gives the…
Mayank Raj
- 121
- 2
- 12
2
votes
1 answer
numpy/pandas: test float64 arrays are equal up to significant digits
I have two pandas data frames in which I store money amounts, i.e. decimal numbers with at most 15 significant decimal digits. Since float64 has a precision of 15 significant decimal digits, this should be lossless.
How do I compare the values of…
ARF
- 7,420
- 8
- 45
- 72
2
votes
2 answers
Float24 (24 bit floating point) to Hex?
I'm using float 24 bit to store a floating point value in a compiler MRK III from NXP. It stores the 24 bit float value as 3 byte Hex in Data memory.
Now when I'm using IEEE 754 float point conversion to retrieve the number back from binary to…
PatrickDew
- 71
- 3
- 10
2
votes
0 answers
Working with high precision floats in webgl
I am working with very high precision floats and i'm adding them to my buffer as vertices.
I have this 2 points that i'm adding to my buffer
{ x: 0.100000001, y: 0 }
{ x: 0.1, y: 0 }
I drew the buffer as points and i see one point.
I checked the…
Raziza O
- 1,560
- 1
- 17
- 37
2
votes
2 answers
PHP Reliable Calculating of Cube Root
It seems like a no brainer but I am having trouble to get this solved:
I would like to calculate a level based on given experience points (exp). Therefore I use the cube root formula and round down to the next whole number. The next level is reached…
stot
- 1,036
- 10
- 20
2
votes
1 answer
Why are certain doubles automatically rounded when printed to the console?
Say I have the following code:
double factor;
double num = 4.35;
BigDecimal y = new BigDecimal(num);
BigDecimal n = new BigDecimal(factor);
BigDecimal asBigDecimal = y.multiply(n);
double asDouble = num * factor;
System.out.println("Double: " +…
null
- 2,060
- 3
- 23
- 42
2
votes
2 answers
Java program to add up the reciprocals of the integers up to number n
I tried to do it but i'm getting 1.0 as answer every time i run it. I'm not being able to find out what's wrong please help me. Here are the codes:
import java.util.Scanner;
public class Number23 {
public static void main(String[] args) {
…
Manisha Singh Sanoo
- 919
- 1
- 13
- 42
2
votes
1 answer
Precision error in numpy.var
I am trying the following code for estimating the variance in a sample, and compare it to the numpy.var implementation.
import numpy as np
def rcov(xj, (i, Mi, Si)):
j = i + 1
Mj = Mi + (xj - Mi) / j
Sj = Si + (i/j) * (xj -…
Paul
- 766
- 9
- 28