Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
48
votes
14 answers

Why do I see a double variable initialized to some value like 21.4 as 21.399999618530273?

double r = 11.631; double theta = 21.4; In the debugger, these are shown as 11.631000000000000 and 21.399999618530273. How can I avoid this?
yesraaj
  • 46,370
  • 69
  • 194
  • 251
47
votes
6 answers

Floating point equality

It is common knowledge that one has to be careful when comparing floating point values. Usually, instead of using ==, we use some epsilon or ULP based equality testing. However, I wonder, are there any cases, when using == is perfectly fine? Look at…
geza
  • 28,403
  • 6
  • 61
  • 135
46
votes
7 answers

How to display a number with always 2 decimal points using BigDecimal?

I am using BigDecimal to get some price values. Requirement is something like this, what ever the value we fetch from database, the displayed valued should have 2 decimal points. Eg: fetched value is 1 - should be displayed as 1.00 fetched value…
user1391730
  • 461
  • 1
  • 4
  • 3
46
votes
4 answers

Precision of multiplication by 1.0 and int to float conversion

Is it safe to assume that the condition (int)(i * 1.0f) == i is true for any integer i?
Violet Giraffe
  • 32,368
  • 48
  • 194
  • 335
45
votes
6 answers

What class to use for money representation?

What class should I use for representation of money to avoid most rounding errors? Should I use Decimal, or a simple built-in number? Is there any existing Money class with support for currency conversion that I could use? Any pitfalls that I should…
Esteban Küber
  • 36,388
  • 15
  • 79
  • 97
43
votes
7 answers

Equals operator for zeros (BigDecimal / Double) in Java

A few interesting observations w.r.t equals operator on 0 and 0.0 new Double(0.0).equals(0) returns false, while new Double(0.0).equals(0.0) returns true. BigDecimal.ZERO.equals(BigDecimal.valueOf(0.0)) returns false, while…
Manish Mulani
  • 7,125
  • 9
  • 43
  • 45
42
votes
2 answers

How to perform unittest for floating point outputs? - python

Let's say I am writing a unit test for a function that returns a floating point number, I can do it as such in full precision as per my machine: >>> import unittest >>> def div(x,y): return x/float(y) ... >>> >>> class…
alvas
  • 115,346
  • 109
  • 446
  • 738
41
votes
7 answers

Why can't I get a p-value smaller than 2.2e-16?

I've found this issue with t-tests and chi-squared in R but I assume this issue applies generally to other tests. If I do: a <- 1:10 b <- 100:110 t.test(a,b) I get: t = -64.6472, df = 18.998, p-value < 2.2e-16. I know from the comments that…
arandomlypickedname
  • 1,349
  • 1
  • 11
  • 12
40
votes
13 answers

Is there a reliable way in JavaScript to obtain the number of decimal places of an arbitrary number?

It's important to note that I'm not looking for a rounding function. I am looking for a function that returns the number of decimal places in an arbitrary number's simplified decimal representation. That is, we have the…
Milosz
  • 2,924
  • 3
  • 22
  • 24
39
votes
5 answers

How best to sum up lots of floating point numbers?

Imagine you have a large array of floating point numbers, of all kinds of sizes. What is the most correct way to calculate the sum, with the least error? For example, when the array looks like this: [1.0, 1e-10, 1e-10, ... 1e-10.0] and you add up…
martinus
  • 17,736
  • 15
  • 72
  • 92
39
votes
2 answers

How can I test floating-point equality using chai?

We're using Chai's BDD API to write unit tests. How can we assert floating point equality? For example, if I try to make this assertion to check for a 66⅔% return value: expect(percentage).to.equal(2 / 3 * 100.0); I get this…
Mattie
  • 20,280
  • 7
  • 36
  • 54
38
votes
3 answers

sine result depends on C++ compiler used

I use the two following C++ compilers: cl.exe : Microsoft (R) C/C++ Optimizing Compiler Version 19.00.24210 for x86 g++ : g++ (Ubuntu 5.2.1-22ubuntu2) 5.2.1 20151010 When using the built-in sine function, I get different results. This is not…
Nicolas
  • 1,812
  • 3
  • 19
  • 43
38
votes
2 answers

What is a bad, decent, good, and excellent F1-measure range?

I understand F1-measure is a harmonic mean of precision and recall. But what values define how good/bad a F1-measure is? I can't seem to find any references (google or academic) answering my question.
KubiK888
  • 4,377
  • 14
  • 61
  • 115
37
votes
11 answers

How to manually parse a floating point number from a string

Of course most languages have library functions for this, but suppose I want to do it myself. Suppose that the float is given like in a C or Java program (except for the 'f' or 'd' suffix), for example "4.2e1", ".42e2" or simply "42". In general, we…
Thomas
  • 174,939
  • 50
  • 355
  • 478
1 2 3
…
99 100