Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
18
votes
4 answers

Rails: Cannot add :precision or :scale options with change_column in a migration?

This seems to have been asked before: rails decimal precision and scale But when running a change_column migration for :precision or :scale they don't actually affect the schema or database, but db:migrate runs without errors. My migration file…
Joshua Pinter
  • 45,245
  • 23
  • 243
  • 245
17
votes
2 answers

Pytorch getting RuntimeError: Found dtype Double but expected Float

I am trying to implement a neural net in PyTorch but it doesn't seem to work. The problem seems to be in the training loop. I've spend several hours into this but can't get it right. Please help, thanks. I haven't added the data preprocessing…
Sanskar Kumar
  • 363
  • 2
  • 5
  • 10
17
votes
1 answer

How to use Gcc 4.6.0 libquadmath and __float128 on x86 and x86_64

I have medium size C99 program which uses long double type (80bit) for floating-point computation. I want to improve precision with new GCC 4.6 extension __float128. As I get, it is a software-emulated 128-bit precision math. How should I convert my…
osgx
  • 90,338
  • 53
  • 357
  • 513
17
votes
5 answers

Why do I need 17 significant digits (and not 16) to represent a double?

Can someone give me an example of a floating point number (double precision), that needs more than 16 significant decimal digits to represent it? I have found in this thread that sometimes you need up to 17 digits, but I am not able to find an…
Ondřej Čertík
  • 780
  • 8
  • 18
17
votes
4 answers

How do I see high-precision query times in mysql command line?

I'm working through some optimization work, and I've noticed that in some mysql dumps people post in articles and questions (which I cannot find again now that I'm actually looking), there are high-precision execution times (0.05985215 sec instead…
Ben Dauphinee
  • 4,061
  • 8
  • 40
  • 59
17
votes
4 answers

How to get largest possible precision? (Python - Decimal)

I'm using the Decimal class for operations that requires precision. I would like to use 'largest possible' precision. With this, I mean as precise as the system on which the program runs can handle. To set a certain precision it's simple: import…
Hades
  • 865
  • 2
  • 11
  • 28
17
votes
10 answers

PI and accuracy of a floating-point number

A single/double/extended-precision floating-point representation of Pi is accurate up to how many decimal places?
Hrushikesh
17
votes
1 answer

What is the most accurate method in python for computing the minimum norm solution or the solution obtained from the pseudo-inverse?

My goal is to solve: Kc=y with the pseudo-inverse (i.e. minimum norm solution): c=K^{+}y such that the model is (hopefully) high degree polynomial model f(x) = sum_i c_i x^i. I am specially interested in the underdetermined case where we have more…
Charlie Parker
  • 5,884
  • 57
  • 198
  • 323
17
votes
5 answers

"new BigDecimal(13.3D)" results in imprecise "13.3000000000000007105.."?

How is it that Java's BigDecimal can be this painful? Double d = 13.3D; BigDecimal bd1 = new BigDecimal(d); BigDecimal bd2 = new BigDecimal(String.valueOf(d)); System.out.println("RESULT 1: "+bd1.toString()); System.out.println("RESULT 2:…
Damo
  • 11,410
  • 5
  • 57
  • 74
17
votes
3 answers

Is there a way to get the "significant figures" of a decimal?

Update OK, after some investigation, and thanks in big part to the helpful answers provided by Jon and Hans, this is what I was able to put together. So far I think it seems to work well. I wouldn't bet my life on its total correctness, of…
Dan Tao
  • 125,917
  • 54
  • 300
  • 447
17
votes
4 answers

How to determine the max precision for double

I am trying to determine what the maximum precision for a double is. In the comments for the accepted answer in this link Retain precision with double in Java @PeterLawrey states max precision in 15. How do you determine this ?
Shivam Sinha
  • 4,924
  • 7
  • 43
  • 65
17
votes
4 answers

Is there a floating point literal suffix in C++ to make a number double precision?

I'm currently working on a C++ project which does numerical calculations. The vast, vast majority of the code uses single precision floating point values and works perfectly fine with that. Because of this I use compiler flags to make basic floating…
user1637052
  • 173
  • 1
  • 1
  • 5
17
votes
2 answers

GNU Octave, round a number to units precision

In GNU Octave version 3.4.3 I want to round a matrix to 2 units precision on the contents of a matrix like this. mymatrix=[1.1234567, 2.12345; 3.1234567891, 4.1234]; disp(mymatrix); This prints: 1.1235 2.1235 3.1235 4.1234 As you can see, the…
Eric Leschinski
  • 146,994
  • 96
  • 417
  • 335
16
votes
3 answers

Java Mutable BigInteger Class

I am doing calculations with BigIntegers that uses a loop that calls multiply() about 100 billion times, and the new object creation from the BigInteger is making it very slow. I was hoping somebody had written or found a MutableBigInteger class. I…
Fractaly
  • 834
  • 2
  • 10
  • 25
16
votes
0 answers

std::cos gives different result when run with valgrind

I've discovered an issue impacting several unit tests at my work, which only happens when the unit tests are run with valgrind, in that the value returned from std::cos and std::sin are different for identical inputs depending on if the unit test is…
Yattabyte
  • 1,280
  • 14
  • 28
1 2 3
…
99 100