Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
26
votes
3 answers

std::fmod abysmal double precision

fmod(1001.0, 0.0001) is giving 0.00009999999995, which seems like a very low precision (10-5), given the expected result of 0. According to cppreference, fmod() is implementable using remainder(), but remainder(1001.0, 0.0001) gives…
rustyx
  • 80,671
  • 25
  • 200
  • 267
26
votes
3 answers

How to convert to double with 2 precision - string after dot?

I want to convert this string: 0.55000000000000004 to this double: 0.55. How to do that?
Furkan Gözükara
  • 22,964
  • 77
  • 205
  • 342
26
votes
6 answers

What is the standard (or best supported) big number (arbitrary precision) library for Lua?

I'm working with large numbers that I can't have rounded off. Using Lua's standard math library, there seem to be no convenient way to preserve precision past some internal limit. I also see there are several libraries that can be loaded to work…
Jon 'links in bio' Ericson
  • 20,880
  • 12
  • 98
  • 148
26
votes
2 answers

Why it is necessary to set precision for the fragment shader?

I learn WebGL. Next shaders work fine: // vertex.shader // precision mediump float; attribute vec4 a_Position; attribute float a_PointSize; void main(){ gl_Position = a_Position; gl_PointSize = a_PointSize; } and // fragment.shader precision…
Andrey Bushman
  • 11,712
  • 17
  • 87
  • 182
26
votes
3 answers

QString:number with maximum 2 decimal places without trailing zero

I have a division like this: number / 1000.0 Sometimes it gives answers like 96.0000000001, sometimes the division works as expected. I want to limit my number to a maximum of two decimal places and without trailing zeros. If it's 96.5500000001 it…
Neaţu Ovidiu Gabriel
  • 833
  • 3
  • 10
  • 20
25
votes
3 answers

Convert float to double loses precision but not via ToString

I have the following code: float f = 0.3f; double d1 = System.Convert.ToDouble(f); double d2 = System.Convert.ToDouble(f.ToString()); The results are equivalent to: d1 = 0.30000001192092896; d2 = 0.3; I'm curious to find out why this is?
JoshG
  • 775
  • 1
  • 7
  • 19
25
votes
2 answers

Does precision change slightly during casting int to double in Java?

I have read this - Why Are Floating Point Numbers Inaccurate? So, sometimes precision can be changed in floating point number because of its representation style (scientific notation with an exponent and a mantissa). But if I cast an integer value…
Uzzal Podder
  • 2,925
  • 23
  • 26
25
votes
6 answers

fopen / fopen_s and writing to files

I'm using fopen in C to write the output to a text file. The function declaration is (where ARRAY_SIZE has been defined earlier): void create_out_file(char file_name[],long double *z1){ FILE *out; int i; if((out = fopen(file_name,…
yCalleecharan
  • 4,656
  • 11
  • 56
  • 86
24
votes
4 answers

Changes to Math.Exp or double implementation in .net 4.5.2

If I run the statement Math.Exp(113.62826122038274).ToString("R") on a machine with .net 4.5.1 installed, then I get the answer 2.2290860617259248E+49 However, if I run the same command on a machine with .net framework 4.5.2 installed, then I get…
Fergus Bown
  • 1,666
  • 9
  • 16
24
votes
3 answers

float and double datatype is good to store latitude and longitude?

I am storing latitude and longitude at the server side (JAVA Platform). To store those values, I am using float and double datatype at the server side. I came to know that float and double are not a recommended primitive datatypes (which is not…
ArunRaj
  • 1,780
  • 2
  • 26
  • 48
23
votes
4 answers

Representing integers in doubles

Can a double (of a given number of bytes, with a reasonable mantissa/exponent balance) always fully precisely hold the range of an unsigned integer of half that number of bytes? E.g. can an eight byte double fully precisely hold the range of numbers…
user82238
23
votes
1 answer

Using math.isclose function with values close to 0

As we know, due to the binary representation of numbers, this expression evaluates to False (at least in Python): 0.2 + 0.4 == 0.6 In order to be able to check for equality within numerical errors, the module math offers isclose: import…
steffen
  • 8,572
  • 11
  • 52
  • 90
23
votes
1 answer

sampling multinomial from small log probability vectors in numpy/scipy

Is there a function in numpy/scipy that lets you sample multinomial from a vector of small log probabilities, without losing precision? example: # sample element randomly from these log probabilities l = [-900, -1680] the naive method fails because…
lgd
  • 1,472
  • 5
  • 17
  • 35
23
votes
5 answers

C# float to decimal conversion

Any smart way to convert a float like this: float f = 711989.98f; into a decimal (or double) without losing precision? I've tried: decimal d = (decimal)f; decimal d1 = (decimal)(Math.Round(f,2)); decimal d2 = Convert.ToDecimal(f);
Adrian4B
  • 233
  • 1
  • 3
  • 5
23
votes
9 answers

Why converting from float to double changes the value?

I've been trying to find out the reason, but I couldn't. Can anybody help me? Look at the following example. float f = 125.32f; System.out.println("value of f = " + f); double d = (double) 125.32f; System.out.println("value of d = " + d); This is…
arthursfreire
  • 727
  • 1
  • 7
  • 17
1 2 3
…
99 100