Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
2
votes
0 answers

Is This Implementation of Logsumexp Numerically Optimal?

The logsumexp algorithm is a technique for calculating the following expression: log( sum( exp( v ) ) ), where the exp function is applied element-wise to the vector v. The typical implementation of the algorithm is the straightforward…
Sean Lake
  • 578
  • 2
  • 8
  • 14
2
votes
1 answer

Floating-point precision when dealing with trigonometric functions

I have a Java class that deals with a lot of trigonometric functions using java.lang.Math. The following code determines the average direction of 0 degrees and 180 degrees. The result should be NaN, since 0 degrees and 180 degrees are in opposite…
null
  • 2,060
  • 3
  • 23
  • 42
2
votes
2 answers

Java BigDecimal data converting to opposite sign long

According to the Java 7 documentation, the method longValue from class java.math.BigDecimal can return a result with the opposite sign. Converts this BigDecimal to a long. This conversion is analogous to the narrowing primitive conversion from…
dinhokz
  • 895
  • 15
  • 36
2
votes
2 answers

Enforce precision in decimal python

In some environments, exact decimals (numerics, numbers...) are defined with scale and precision, with scale being all significant numbers, and precision being those right of the decimal point. I want to use python's decimal implementation to raise…
Dschoni
  • 3,714
  • 6
  • 45
  • 80
2
votes
3 answers

Groovy - confusion double vs float

I'm using below two statements :- double foo = 20.00 float bar = 20.00 println foo == bar And double foo = 20.01 float bar = 20.01 println foo == bar It gives the output as :- true false Can anyone know what makes difference between these two…
Saurabh Gaur
  • 23,507
  • 10
  • 54
  • 73
2
votes
1 answer

What is the difference between auc_score and precision_at_k for recommendation engines (LightFM)

I'm currently trying to build a recommendation engine using the Python LightFM library. My input is a sparse matrix with shape (n_users, n_items), where each cell value within the matrix represents the number of interactions a user has had with the…
andre
  • 41
  • 6
2
votes
1 answer

Lack of precision in Andres drawn circle

I wrote a little program which draws a circular chromatic gradation using Andres' algorithm. Here is the result of an execution : If the final-user gives the program a shift, in radians, it will use the latter to begin the drawing of the circle at…
JarsOfJam-Scheduler
  • 2,809
  • 3
  • 31
  • 70
2
votes
1 answer

Logarithm to integer base of integer

I would like to find the upward rounded value of logarithm of integer n to integer base b. In code: result = int(ceil(log(n, b))) The problem is that sometimes the value cannot be represented exactly in floating point, overestimating the result.…
Ecir Hana
  • 10,864
  • 13
  • 67
  • 117
2
votes
3 answers

Speed up random number generation in MATLAB

Is there any way to generate pseudo-random numbers to less precision and thus speed the process up? Another thing is that I know it saves time if random numbers are generated all at once (e.g. rand(100,1000)), instead of one by one. Could someone…
moon1234
  • 123
  • 3
2
votes
1 answer

Extra precision required for the implementation of trig functions with BigDecimal

Introduction I am interested in writing math functions for BigDecimal (actually, also for my own BigDecimal type written in Delphi, but that is irrelevant here -- in this question, I use Java's BigDecimal because more people know it and my…
Rudy Velthuis
  • 28,387
  • 5
  • 46
  • 94
2
votes
1 answer

Conversion of decimal floating point numbers to binary and back

My question, in short is, why does rounding error in floats only show up after calculations and not for storage of literals? What I mean is this - I know about the issues that arise due to rounding error in floats when converting from decimal to…
Rohan Saxena
  • 3,133
  • 2
  • 16
  • 34
2
votes
1 answer

problems with the erf/erfc function in matlab

I am having problems with the accuracy of Matlab's erf/erfc functions. As we know, erf(x) is only equal to 1 if x is infinity. However, in Matlab, I am surprised that erf(6) is already equal to 1 and 6 is not even considerably large! erfc(x) is a…
user1237300
  • 231
  • 2
  • 11
2
votes
0 answers

Floating point relative error bounds, clarification needed

I'm reading David Goldberg's What Every Computer Scientist Should Know About Floating Point Arithmetic paper, and I'm confused by one of the inequalities (2): (1/2)B^-p <= (1/2)ulp <= (B/2)B^-p I follow the reasoning behind the left hand side and…
Joseph Garvin
  • 20,727
  • 18
  • 94
  • 165
2
votes
2 answers

Quadruple precision for abs and sign

I am trying to change the precision of the abs and sign with gfortran (gcc version 5.3.1 and x86_64-linux-gnu). I know I can change to dabs and dsign to have double precision but what about quad precision, is it possible? For sqrt(x) for instance,…
Panichi Pattumeros PapaCastoro
  • 1,034
  • 3
  • 20
  • 44
2
votes
2 answers

Python/Matlab - Taking rank of matrix in quad precision or more

I have a 14x14 matrix of which I'm trying to take the rank. The problem is that it has a high condition number so using double precision my matrix is not full rank. I know that it should be, so I'm trying to take the rank in higher precision. So…
0_o
  • 333
  • 3
  • 9
1 2 3
…
99 100