Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
22
votes
1 answer

Correcting "summary" in R with appropriate # of digits of precision

A simple question on a simple seemingly innocent function: summary. Until I saw results for Min and Max that were outside the range of my data, I was unaware that summary has a digits argument to specify precision of the output results. My…
Iterator
  • 20,250
  • 12
  • 75
  • 111
22
votes
5 answers

Scikit: calculate precision and recall using cross_val_score function

I'm using scikit to perform a logistic regression on spam/ham data. X_train is my training data and y_train the labels('spam' or 'ham') and I trained my LogisticRegression this way: classifier = LogisticRegression() classifier.fit(X_train,…
Anil Narassiguin
  • 541
  • 1
  • 4
  • 10
21
votes
4 answers

Double precision is different in different languages

I'm experimenting with the precision of a double value in various programming languages. My programs main.c #include int main() { for (double i = 0.0; i < 3; i = i + 0.1) { printf("%.17lf\n", i); } return…
Jaysmito Mukherjee
  • 1,467
  • 2
  • 10
  • 29
20
votes
4 answers

Dealing with very small numbers in R

I need to calculate a list of very small numbers such as (0.1)^1000, 0.2^(1200), and then normalize them so they will sum up to one i.e. a1 = 0.1^1000, a2 = 0.2^1200 And I want to calculate a1' = a1/(a1+a2), a2'=a2(a1+a2). I'm running into…
dan12345
  • 1,594
  • 4
  • 20
  • 30
20
votes
2 answers

why in matlab sin(pi) is not exact but sin(pi/2) is exact?

I have a problem in calculation with matlab. I know that "pi" is a floating number and is not exact. So, in matlab sin(pi) is not exactly zero. My question is if "pi" is not exact then why sin(pi/2) is exactly equal 1. sin(pi) --> is not exact…
Nafiseh Eftekharian
  • 201
  • 1
  • 4
20
votes
1 answer

Parse unix timestamp with decimal from float using time.Unix()?

This is pretty straightforward, but I couldn't find a response and figured that others might have the same question. I have a Unix timestamp as a float, that includes a decimal value for fractions of seconds. What's the conversion factor to pass…
Ben Guild
  • 4,881
  • 7
  • 34
  • 60
20
votes
7 answers

Exact decimal datatype for C++?

PHP has a decimal type, which doesn't have the "inaccuracy" of floats and doubles, so that 2.5 + 2.5 = 5 and not 4.999999999978325 or something like that. So I wonder if there is such a data type implementation for C or C++?
dtech
  • 47,916
  • 17
  • 112
  • 190
19
votes
2 answers

How many digits can float8, float16, float32, float64, and float128 contain?

Numpy's dtype documentation only shows "x bits exponent, y bits mantissa" for each float type, but I couldn't translate that to exactly how many digits before/after the decimal point. Is there any simple formula/table to look this up in?
mathguy
  • 1,450
  • 1
  • 16
  • 33
19
votes
4 answers

How to convert float to string

I read a float from a file and will have to convert it to string. My problem here is that I am unsure of how many digits will be there after the decimal. I need to take the float exactly and convert it to string. For ex: 1.10 should be converted to…
Nagireddy Hanisha
  • 1,290
  • 4
  • 17
  • 39
19
votes
3 answers

Why does a C# System.Decimal remember trailing zeros?

Is there a reason that a C# System.Decimal remembers the number of trailing zeros it was entered with? See the following example: public void DoSomething() { decimal dec1 = 0.5M; decimal dec2 = 0.50M; Console.WriteLine(dec1); …
Robert Davey
  • 507
  • 1
  • 4
  • 13
19
votes
5 answers

Set precision for a float number in PHP

I get a number from database and this number might be either float or int. I need to set the decimal precision of the number to 3, which makes the number not longer than (regarding decimals) 5.020 or 1518845.756. Using PHP round($number,…
Mostafa Talebi
  • 8,825
  • 16
  • 61
  • 105
18
votes
5 answers

losing precision converting from java BigDecimal to double

I am working with an application that is based entirely on doubles, and am having trouble in one utility method that parses a string into a double. I've found a fix where using BigDecimal for the conversion solves the issue, but raises another…
Edward Q. Bridges
  • 16,712
  • 8
  • 35
  • 42
18
votes
3 answers

Why am I losing precision while converting float32 to float64?

While converting a float32 number to float64 precision is being lost in Go. For example converting 359.9 to float64 produces 359.8999938964844. If float32 can be stored precisely why is float64 losing precision? Sample code: package main import ( …
Mayank Patel
  • 8,088
  • 5
  • 55
  • 75
18
votes
2 answers

Invertability of IEEE 754 floating-point division

What is the invertability of the IEEE 754 floating-point division? I mean is it guaranteed by the standard that if double y = 1.0 / x then x == 1.0 / y, i.e. x can be restored precisely bit by bit? The cases when y is infinity or NaN are obvious…
plasmacel
  • 8,183
  • 7
  • 53
  • 101
18
votes
2 answers

Setting minimum number of decimal places for std::ostream precision

Is there a way to set the "minimum" number of decimal places that a std::ostream will output? For example, say I have two unknown double variables that I want to print (values added here for the sake of illustration): double a = 0; double b =…
Phil Boltt
  • 786
  • 6
  • 14
1 2 3
…
99 100