Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
15
votes
4 answers

gcc rounding difference between versions

I'm looking into why a test case is failing The problematic test can be reduced to doing (4.0/9.0) ** (1.0/2.6), rounding this to 6 digits and checking against a known value (as a string): #include #include int main(){ …
dbr
  • 165,801
  • 69
  • 278
  • 343
15
votes
1 answer

How does sklearn select threshold steps in precision recall curve?

I trained a basic FFNN on a example breast cancer dataset. For the results the precision_recall_curve function gives datapoints for 416 different thresholds. My Data contains 569 unique prediction values, as far as I understand the Precision Recall…
Quastiat
  • 1,164
  • 1
  • 18
  • 37
15
votes
2 answers

Can the Precision, Recall and F1 be the same value?

I am currently working on an ML classification problem and I'm computing the Precision, Recall and F1 using the sklearn library's following import and respective code as shown below. from sklearn.metrics import…
Nayantara Jeyaraj
  • 2,624
  • 7
  • 34
  • 63
15
votes
2 answers

tf.round() to a specified precision

tf.round(x) rounds the values of x to integer values. Is there any way to round to, say, 3 decimal places instead?
KOB
  • 4,084
  • 9
  • 44
  • 88
15
votes
3 answers

How does C++ integer division work for limit and negative values?

I am facing some strange results with integer division in C++. I am trying to calculate this: -2147483648 / -1. What I get is 3 different results in 3 different scenarios: int foo(int numerator, int denominator) { int res = numerator /…
Kareem Ergawy
  • 647
  • 1
  • 8
  • 17
15
votes
7 answers

Auto-Interval precision in MS Chart

I'm currently using the charting within .NET using System.Windows.Forms.DataVisualization.Charting.Chart. Thus far it seems very powerful, and works great. However, there is a huge problem in terms of how it is auto-calculating intervals. I use a…
drharris
  • 11,194
  • 5
  • 43
  • 56
15
votes
1 answer

How do you convert to half floats in JavaScript?

I want to be able to use the OES_texture_half_float extension in WebGL and provide my own data but there's no Float16Array in JavaScript. So how do I generate half float data?
gman
  • 100,619
  • 31
  • 269
  • 393
15
votes
2 answers

How can I fix error code C4146 "unary minus operator applied to unsigned type.result still unsigned"?

Data type int's minimum value is -2,147,483,648. So, I typed int val = -2147483648; But, it has an error: unary minus operator applied to unsigned type.result still unsigned How can I fix it?
양기창
  • 169
  • 1
  • 1
  • 3
15
votes
3 answers

What is the purpose of max_digits10 and how is it different from digits10?

I am confused about what max_digits10 represents. According to its documentation, it is 0 for all integral types. The formula for floating-point types for max_digits10 looks similar to int's digits10's.
NoSenseEtAl
  • 28,205
  • 28
  • 128
  • 277
15
votes
2 answers

Matrix multiplication on CPU (numpy) and GPU (gnumpy) give different results

I'm using gnumpy to speed up some computations in training a neural network by doing them on GPU. I'm getting the desired speedup but am a little bit worried about the differences in the results of numpy (cpu) vs gnumpy (gpu). I have the following…
Ottokar
  • 403
  • 1
  • 3
  • 8
15
votes
4 answers

Why is the output of inv() and pinv() not equal in Matlab and Octave?

I have noticed that if A is a NxN matrix and it has the inverse matrix. But what the inv() and pinv() function output is different. - My environment is Win7x64 SP1, Matlab R2012a, Cygwin Octave 3.6.4, FreeMat 4.2 Have a look at the examples from…
myme5261314
  • 462
  • 1
  • 5
  • 14
15
votes
8 answers

Preserving large numbers

I am trying to read a CSV file that has barcodes in the first column, but when R gets it into a data.frame, it converts 1665535004661 to 1.67E+12. Is there a way to preserve this number in an integer format? I tried assigning a class of "double",…
James
  • 1,447
  • 3
  • 16
  • 30
14
votes
2 answers

How to calculate sums in log-space without underflow?

I am trying to calculate log(a + b) given log(a) and log(b). The problem is, log(a) and log(b) are so negative that when I try to calculate a and b themselves, they underflow and I get log(0), which is undefined. For log(a * b) and log(a / b), this…
Dominick Pastore
  • 4,177
  • 2
  • 17
  • 29
14
votes
4 answers

How can I display numbers with higher precision in a MATLAB data cursor?

I have a problem with precision loss. I imported a set of values from a CSV file into MATLAB 7 using the following code: function importfile(fileToRead1) %#IMPORTFILE(FILETOREAD1) %# Imports data from the specified file %# FILETOREAD1: file to…
wishi
  • 7,188
  • 17
  • 64
  • 103
14
votes
1 answer

Why is np.dot imprecise? (n-dim arrays)

Suppose we take np.dot of two 'float32' 2D arrays: res = np.dot(a, b) # see CASE 1 print(list(res[0])) # list shows more digits [-0.90448684, -1.1708503, 0.907136, 3.5594249, 1.1374011, -1.3826287] Numbers. Except, they can change: CASE 1:…
OverLordGoldDragon
  • 1
  • 9
  • 53
  • 101
1 2 3
…
99 100