Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
160
votes
10 answers

Is floating-point math consistent in C#? Can it be?

No, this is not another "Why is (1/3.0)*3 != 1" question. I've been reading about floating-points a lot lately; specifically, how the same calculation might give different results on different architectures or optimization settings. This is a…
BlueRaja - Danny Pflughoeft
  • 84,206
  • 33
  • 197
  • 283
154
votes
3 answers

Why does adding 0.1 multiple times remain lossless?

I know the 0.1 decimal number cannot be represented exactly with a finite binary number (explanation), so double n = 0.1 will lose some precision and will not be exactly 0.1. On the other hand 0.5 can be represented exactly because it is 0.5 = 1/2 =…
icza
  • 389,944
  • 63
  • 907
  • 827
147
votes
4 answers

Controlling number of decimal digits in print output in R

There is an option in R to get control over digit display. For example: options(digits=10) is supposed to give the calculation results in 10 digits till the end of R session. In the help file of R, the definition for digits parameter is as…
Mehper C. Palavuzlar
  • 10,089
  • 23
  • 56
  • 69
140
votes
13 answers

PHP7.1 json_encode() Float Issue

This isn't a question as it is more of a be aware. I updated an application that uses json_encode() to PHP7.1.1 and I was seeing an issue with floats being changed to sometimes extend out 17 digits. According to documentation, PHP 7.1.x started to…
Gwi7d31
  • 1,652
  • 2
  • 12
  • 10
133
votes
5 answers

Double precision floating values in Python?

Are there data types with better precision than float?
kravemir
  • 10,636
  • 17
  • 64
  • 111
125
votes
23 answers

How to properly round-up half float numbers?

I am facing a strange behavior of the round() function: for i in range(1, 15, 2): n = i / 2 print(n, "=>", round(n)) This code prints: 0.5 => 0 1.5 => 2 2.5 => 2 3.5 => 4 4.5 => 4 5.5 => 6 6.5 => 6 I expected the floating values to be…
Delgan
  • 18,571
  • 11
  • 90
  • 141
112
votes
11 answers

In which order should floats be added to get the most precise result?

This was a question I was asked at my recent interview and I want to know (I don't actually remember the theory of the numerical analysis, so please help me :) If we have some function, which accumulates floating-point…
Yippie-Ki-Yay
  • 22,026
  • 26
  • 90
  • 148
110
votes
10 answers

Is it safe to check floating point values for equality to 0?

I know you can't rely on equality between double or decimal type values normally, but I'm wondering if 0 is a special case. While I can understand imprecisions between 0.00000000000001 and 0.00000000000002, 0 itself seems pretty hard to mess up…
Gene Roberts
  • 2,192
  • 3
  • 17
  • 16
110
votes
6 answers

BigDecimal, precision and scale

I'm using BigDecimal for my numbers in my application, for example, with JPA. I did a bit of researching about the terms 'precision' and 'scale' but I don't understand what are they exactly. Can anyone explain me the meaning of 'precision' and…
jpadilladev
  • 1,756
  • 4
  • 16
  • 23
108
votes
7 answers

C# DateTime.Now precision

I just ran into some unexpected behavior with DateTime.UtcNow while doing some unit tests. It appears that when you call DateTime.Now/UtcNow in rapid succession, it seems to give you back the same value for a longer-than-expected interval of time,…
Andy White
  • 86,444
  • 48
  • 176
  • 211
103
votes
6 answers

How many double numbers are there between 0.0 and 1.0?

This is something that's been on my mind for years, but I never took the time to ask before. Many (pseudo) random number generators generate a random number between 0.0 and 1.0. Mathematically there are infinite numbers in this range, but double is…
polygenelubricants
  • 376,812
  • 128
  • 561
  • 623
96
votes
8 answers

PHP - Floating Number Precision

$a = '35'; $b = '-34.99'; echo ($a + $b); Results in 0.009999999999998 What is up with that? I wondered why my program kept reporting odd results. Why doesn't PHP return the expected 0.01?
dcmoody
  • 1,295
  • 1
  • 11
  • 15
93
votes
1 answer

Dividing two integers to produce a float result

Possible Duplicate: Why can't I return a double from two ints being divided My C++ program is truncating the output of my integer devision even when I try and place the output into a float. How can I prevent this whilst keeping those to variables…
jwbensley
  • 10,534
  • 19
  • 75
  • 93
90
votes
12 answers

Endless sine generation in C

I am working on a project which incorporates computing a sine wave as input for a control loop. The sine wave has a frequency of 280 Hz, and the control loop runs every 30 µs and everything is written in C for an Arm Cortex-M7. At the moment we are…
energetic
  • 897
  • 1
  • 5
  • 7
85
votes
7 answers

What range of numbers can be represented in a 16-, 32- and 64-bit IEEE-754 systems?

I know a little bit about how floating-point numbers are represented, but not enough, I'm afraid. The general question is: For a given precision (for my purposes, the number of accurate decimal places in base 10), what range of numbers can be…
Nate Parsons
  • 14,431
  • 13
  • 51
  • 67
1
2
3
…
99 100