Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
32
votes
2 answers

Golang floating point precision float32 vs float64

I wrote a program to demonstrate floating point error in Go: func main() { a := float64(0.2) a += 0.1 a -= 0.3 var i int for i = 0; a < 1.0; i++ { a += a } fmt.Printf("After %d iterations, a = %e\n", i, a) } It…
charliehorse55
  • 1,940
  • 5
  • 24
  • 38
31
votes
3 answers

Why is my arithmetic with a long long int behaving this way?

I'm trying to calculate large integers with the long long datatype but when it gets large enough (2^55), the arithmetic behavior is unpredictable. I am working in Microsoft Visual Studio 2017. In this first case, I am subtracting 2 from the long…
Edvin K
  • 329
  • 3
  • 5
30
votes
4 answers

LocalDateTime.now() has different levels of precision on Windows and Mac machine

When creating a new LocalDateTime using LocalDateTime.now() on my Mac and Windows machine i get a nano precision of 6 on my Mac and a nano precision of 3 on my Windows machine. Both are running jdk-1.8.0-172. Is it possible to limit or increase…
Tobias Marschall
  • 2,355
  • 3
  • 22
  • 40
29
votes
8 answers

Truncate a floating point number without rounding up

I have a floating point number that I want to truncate to 3 places but I don't want to round up. For example, convert 1.0155555555555555 to 1.015 (not 1.016). How would I go about doing this in Ruby?
ab217
  • 16,900
  • 25
  • 74
  • 92
29
votes
2 answers

Why do bean validation Min/Max constraints not support the double type?

Could somebody please explain to me why the JPA supports the double type as a field type, but the bean validation constaints in javax.validation.constraints (i.e. @Min/@Max) do not support it? I know documentation says this is due to rounding…
Manuel Leuenberger
  • 2,327
  • 3
  • 21
  • 27
29
votes
2 answers

What is the precision of long double in C++?

Does anyone know how to find out the precision of long double on a specific platform? I appear to be losing precision after 17 decimal digits, which is the same as when I just use double. I would expect to get more, since double is represented…
Bill the Lizard
  • 398,270
  • 210
  • 566
  • 880
29
votes
6 answers

T-SQL Decimal Division Accuracy

Does anyone know why, using SQLServer 2005 SELECT CONVERT(DECIMAL(30,15),146804871.212533)/CONVERT(DECIMAL (38,9),12499999.9999) gives me 11.74438969709659, but when I increase the decimal places on the denominator to 15, I get a less accurate…
MT.
  • 377
  • 2
  • 4
  • 9
29
votes
2 answers

Python numpy float16 datatype operations, and float8?

when performing math operations on float16 Numpy numbers, the result is also in float16 type number. My question is how exactly the result is computed? Say Im multiplying/adding two float16 numbers, does python generate the result in float32 and…
JonyK
  • 585
  • 2
  • 7
  • 12
29
votes
9 answers

Loss of precision - int -> float or double

I have an exam question I am revising for and the question is for 4 marks. "In java we can assign a int to a double or a float". Will this ever lose information and why? I have put that because ints are normally of fixed length or size - the…
stan
  • 433
  • 1
  • 5
  • 7
29
votes
3 answers

Why does the Java increment operator allow narrowing operations without explicit cast?

Possible Duplicate: Java += operator In Java, this is not valid (doesn't compile), as expected: long lng = 0xffffffffffffL; int i; i = 5 + lng; //"error: possible loss of magnitude" But this is perfectly fine (?!) long lng =…
Cristian Diaconescu
  • 34,633
  • 32
  • 143
  • 233
28
votes
5 answers

How to print a double with full precision on iOS?

Test case: NSLog(@"%f", M_PI); NSLog(@"%@", [NSString stringWithFormat:@"%f", M_PI]); NSLog(@"%@", [NSNumber numberWithDouble:M_PI]); Results: 3.141593 3.141593 3.141592653589793 Conclusions: 1) Printing via NSLog() or [NSString…
Ariel Malka
  • 15,697
  • 6
  • 31
  • 33
28
votes
9 answers

How to 'cout' the correct number of decimal places of a double value?

I need help on keeping the precision of a double. If I assign a literal to a double, the actual value was truncated. int main() { double x = 7.40200133400; std::cout << x << "\n"; } For the above code snippet, the output was 7.402 Is there…
roxrook
  • 13,511
  • 40
  • 107
  • 156
27
votes
8 answers

Ensuring C++ doubles are 64 bits

In my C++ program, I need to pull a 64 bit float from an external byte sequence. Is there some way to ensure, at compile-time, that doubles are 64 bits? Is there some other type I should use to store the data instead? Edit: If you're reading this…
Whatsit
  • 10,227
  • 11
  • 42
  • 41
27
votes
4 answers

High precision floating point numbers in Haskell?

I know Haskell has native data types which allow you to have really big integers so things like >> let x = 131242358045284502395482305 >> x 131242358045284502395482305 work as expected. I was wondering if there was a similar "large precision…
MYV
  • 4,294
  • 6
  • 28
  • 24
27
votes
8 answers

Next higher/lower IEEE double precision number

I am doing high precision scientific computations. In looking for the best representation of various effects, I keep coming up with reasons to want to get the next higher (or lower) double precision number available. Essentially, what I want to do…
Mark T
  • 3,464
  • 5
  • 31
  • 45
1 2 3
…
99 100