Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
14
votes
3 answers

MATLAB precision

How can I implement quadruple precision (128 bit arithmetic) in MATLAB while solving a Matrix eigenvalue problem ? I am trying to solve a linear stability analysis problem for incompressible plane Couette flow and the the default 64 bit precision of…
Dev
  • 143
  • 1
  • 1
  • 6
14
votes
3 answers

Can C# store more precise data than doubles?

double in C# don't hold enough precision for my needs. I am writing a fractal program, and after zooming in a few times I run out of precision. I there a data type that can hold more precise floating-point information (i.e more decimal places) than…
simonalexander2005
  • 4,338
  • 4
  • 48
  • 92
14
votes
0 answers

How to solve Cholesky decomposition error in Tensorflow caused by low precision datatype tf.float32?

It seems that tensorflow only supports tf.float32 for training neural network, and this causes a Cholesky decomposition issues in my algorithm. The following code is a part of my computational graph, where X_latent is a tensor to be passed in, and…
user25764
  • 141
  • 1
  • 4
14
votes
1 answer

ProgrammingError: column "X" is of type double precision but expression is of type numeric[]

I am trying to program an API with Django Rest Framework and Psycopg2 library. This API works with PostgreSQL database. I need to store 2 float array fields in this db because the API works with 2 sensors which stream data to an Android App, and…
Nacho Díaz Reyes
  • 141
  • 1
  • 3
14
votes
3 answers

logistic / sigmoid function implementation numerical precision

in scipy.special.expit, logistic function is implemented like the following: if x < 0 a = exp(x) a / (1 + a) else 1 / (1 + exp(-x)) However, I have seen implementations in other languages/frameworks that simply do 1 / (1 +…
colinfang
  • 20,909
  • 19
  • 90
  • 173
14
votes
2 answers

Machine Epsilon in Python

A manual that I am currently studying (I am a newbie) says: "Numbers which differ by less than machine epsilon are numerically the same" With Python, machine epsilon for float values can be obtained by typing eps = numpy.finfo(float).eps Now, If…
Charlie
  • 286
  • 1
  • 2
  • 9
14
votes
6 answers

Why double can store bigger numbers than unsigned long long?

The question is, I don't quite get why double can store bigger numbers than unsigned long long. Since both of them are 8 bytes long, so 64 bits. Where in unsigned long long, all 64 bits are used in order to store a value, on the other hand double…
denis631
  • 1,765
  • 3
  • 17
  • 38
14
votes
2 answers

Math.log2 precision has changed in Chrome

I've written a JavaScript program that calculates the depth of a binary tree based on the number of elements. My program has been working fine for months, but recently I've found a difference when the web page is viewed in Chrome vs Firefox. In…
Stefan Musarra
  • 1,429
  • 14
  • 16
14
votes
10 answers

2.9999999999999999 >> .5?

I heard that you could right-shift a number by .5 instead of using Math.floor(). I decided to check its limits to make sure that it was a suitable replacement, so I checked the following values and got the following results in Google Chrome: 2.5 >>…
Zen
  • 167
  • 1
  • 5
14
votes
4 answers

Fast double to string conversion with given precision

I need to convert double to string with given precision. String.format("%.3f", value) (or DecimalFormat) does the job, but benchmarks show that it is slow. Even Double.toString conversion takes about 1-3 seconds to convert 1 million numbers on my…
valodzka
  • 5,535
  • 4
  • 39
  • 50
13
votes
2 answers

Difference between double- precision data type and numeric data type

What is the difference between double-precision data type and numeric data type in R programming?
Devyani Balyan
  • 175
  • 1
  • 1
  • 10
13
votes
2 answers

Decimal precision of floats

equivalent to log10(2^24) ≈ 7.225 decimal digits Wikipedia Precision: 7 digits MSDN 6 std::numeric_limits::digits10 Why numeric_limits return 6 here? Both Wikipedia and MSDN report that floats have 7 decimal digits of precision.
liori
  • 40,917
  • 13
  • 78
  • 105
13
votes
7 answers

Determine the decimal precision of an input number

We have an interesting problem were we need to determine the decimal precision of a users input (textbox). Essentially we need to know the number of decimal places entered and then return a precision number, this is best illustrated with…
Quinten
  • 131
  • 1
  • 1
  • 3
13
votes
5 answers

Java BigDecimal trigonometric methods

I am developing a mathematical parser which is able to evaluate String like '5+b*sqrt(c^2)'. I am using ANTLR for the parsing and make good progress. Now I fell over the Java class BigDecimal and thought: hey, why not thinking about precision…
Marco
  • 133
  • 1
  • 1
  • 5
13
votes
4 answers

sizeof long double and precision not matching?

Consider the following C code: #include int main(int argc, char* argv[]) { const long double ld = 0.12345678901234567890123456789012345L; printf("%lu %.36Lf\n", sizeof(ld), ld); return 0; } Compiled with gcc 4.8.1 under…
Vincent
  • 57,703
  • 61
  • 205
  • 388
1 2 3
…
99 100