Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
37
votes
5 answers

Half-precision floating-point in Java

Is there a Java library anywhere that can perform computations on IEEE 754 half-precision numbers or convert them to and from double-precision? Either of these approaches would be suitable: Keep the numbers in half-precision format and compute…
finnw
  • 47,861
  • 24
  • 143
  • 221
37
votes
2 answers

What is the correct/standard way to check if difference is smaller than machine precision?

I often end up in situations where it is necessary to check if the obtained difference is above machine precision. Seems like for this purpose R has a handy variable: .Machine$double.eps. However when I turn to R source code for guidelines about…
Karolis Koncevičius
  • 9,417
  • 9
  • 56
  • 89
37
votes
5 answers

How to check if float pandas column contains only integer numbers?

I have a dataframe df = pd.DataFrame(data=np.arange(10),columns=['v']).astype(float) How to make sure that the numbers in v are whole numbers? I am very concerned about rounding/truncation/floating point representation errors
00__00__00
  • 4,834
  • 9
  • 41
  • 89
37
votes
5 answers

PHP and unit testing assertions with decimals

I have a method that returns a float like 1.234567890.I want to test that it really does so. However, it seems that this returned float has different precision on different platforms so how do I assert that the returned value is 1.23456789? If I…
Tower
  • 98,741
  • 129
  • 357
  • 507
37
votes
3 answers

What does "real*8" mean?

The manual of a program written in Fortran 90 says, "All real variables and parameters are specified in 64-bit precision (i.e. real*8)." According to Wikipedia, single precision corresponds to 32-bit precision, whereas double precision corresponds…
Andrew
  • 1,499
  • 9
  • 25
  • 37
35
votes
5 answers

Show two digits after decimal point in c++

Similar topic is already discussed in the forum. But I have some different problem in following code: double total; cin >> total; cout << fixed << setprecision(2) << total; If I give input as 100.00 then program prints just 100 but not 100.00 How…
DSKVP
  • 663
  • 2
  • 11
  • 19
34
votes
3 answers

C# double to decimal precision loss

I have a double "138630.78380386264" and I want to convert it to a decimal, however when I do so I do it either by casting or by using Convert.ToDecimal() and I lose precision. What's going on? Both decimal and double can hold this number: double…
GreyCloud
  • 3,030
  • 5
  • 32
  • 47
34
votes
10 answers

Find angle between hour and minute hands in an analog clock

I was given this interview question recently: Given a 12-hour analog clock, compute in degree the smaller angle between the hour and minute hands. Be as precise as you can. I'm wondering what's the simplest, most readable, most precise algorithm…
polygenelubricants
  • 376,812
  • 128
  • 561
  • 623
34
votes
3 answers

How to convert milliseconds to seconds with precision

I want to convert milliseconds to seconds (for example 1500ms to 1.5s, or 500ms to 0.5s) with as much precision as possible. Double.parseDouble(500 / 1000 + "." + 500 % 1000); isn't the best way to do it: I'm looking for a way to get the remainder…
Keir Nellyer
  • 913
  • 5
  • 12
  • 20
33
votes
5 answers

How can I make sure a float will always be rounded up with PHP?

I want to make sure a float in PHP is rounded up if any decimal is present, without worrying about mathematical rounding rules. This function would work as follows: 1.1 to 2 1.2 to 2 1.9 to 2 2.3 to 3 2.8 to 3 I know the round() function exists but…
Sotiris
  • 38,986
  • 11
  • 53
  • 85
33
votes
9 answers

ArithmeticException thrown during BigDecimal.divide

I thought java.math.BigDecimal is supposed to be The Answer™ to the need of performing infinite precision arithmetic with decimal numbers. Consider the following snippet: import java.math.BigDecimal; //... final BigDecimal one =…
polygenelubricants
  • 376,812
  • 128
  • 561
  • 623
33
votes
2 answers

Haskell: Force floats to have two decimals

Using the following code snippet: (fromIntegral 100)/10.00 Using the Haskell '98 standard prelude, how do I represent the result with two decimals? Thanks.
Anders
  • 856
  • 1
  • 7
  • 14
33
votes
5 answers

Set back default floating point print precision in C++

I want to control the precision for a double during a comparison, and then come back to default precision, with C++. I intend to use setPrecision() to set precision. What is then syntax, if any, to set precision back to default? I am doing something…
kiriloff
  • 25,609
  • 37
  • 148
  • 229
32
votes
2 answers

C# Converting 20 digit precision double to string and back again

In C#. I have a double (which I've extracted from a database) that has 20 digit precision. In Visual Studio (using QuickWatch) I can see the value of the double to be = 0.00034101243963859839. I want to display this value in a textbox and then…
JohnM
32
votes
5 answers

How to Java String.format with a variable precision?

I'd like to allow the user to vary the precision of a String generated from a double. Right now, I'm trying something like String foo = String.format("%.*f\n", precision, my_double); However, I receive a java.util.UnknownFormatConversionException.…
Willi Ballenthin
  • 6,444
  • 6
  • 38
  • 52
1 2 3
…
99 100