Stack Overflow
Stack Overflow

Questions tagged [double-precision]

Anything related to double-precision floating-point arithmetic and data-types. Often used with reference to IEEE754 double-precision floating-point representation.

240 questions
2
votes
6 answers

Handling loss of precision in subtracting two doubles that are close to each other

I have a project to do where we are to solve the matrix equation AX=B for x, given that A is a tridiagonal matrix. I did this project in C++, got the program to produce the right Matrix X, but when trying to report the error back to the user, A*X-B,…
Mike Warren
  • 3,796
  • 5
  • 47
  • 99
2
votes
2 answers

postgres query on double precision column not returning correct results

I have a double precision column in my database called "position." If I have a row with a value in that column that is: 0.564593301435407 And I have a query that has a WHERE statement like the following: WHERE "position" >=…
Steph Rose
  • 2,126
  • 3
  • 23
  • 35
2
votes
4 answers

Differentiate between single and double precision

Python's float data type really uses double precision (64bit). However, for my specific implementation (that transmits typetagged values via OSC) I would to like differentiate between values that can be represented as (32bit) single precision floats…
umläute
  • 28,885
  • 9
  • 68
  • 122
2
votes
2 answers

why is there significant double precision difference between Matlab and Mathematica?

I created a random double precision value in Matlab by x = rand(1,1); then display all possible digits of x by vpa(x,100) and obtain: 0.2238119394911369 7971853298440692014992237091064453125 I save x to a .mat file, and import it into…
LCFactorization
  • 1,652
  • 1
  • 25
  • 35
2
votes
1 answer

Double precision of a number

If we assume that a variable x and y can be represented in double precision without an error. Would it be better to writte x^2 - y^2 or just (x + y)(x-y)? I thought about this problem, and think that x^2 - y^2 should be more accurate, because of the…
Carol.Kar
  • 4,581
  • 36
  • 131
  • 264
2
votes
1 answer

MPI_REDUCE precision problems

I am encountering precision problems with MPI_REDUCE() in fortran. I have tested two methods of summing double precision numbers stored on each node. The MPI_REDUCE() line I use is call…
DJames
  • 571
  • 3
  • 17
2
votes
2 answers

C++ precision - behaviour of setprecision

As I understand it the setprecision function specifies the minimal precision but when I run the following code I get only 3 numbers after the decimal point: int main() { double a = 123.4567890; double b = 123.4000000; std::cout <<…
SadStudent
  • 287
  • 1
  • 4
  • 16
2
votes
1 answer

kCFNumberFormatter round differences

I found an interesting question what's not described on the internet, even is misleading in the apple documentation. What are the differences between:…
flatronka
  • 1,061
  • 25
  • 51
2
votes
1 answer

CUDA double precision and number of registers per thread

I am having an error while executing the kernel too many resources requested for launch I checked online for any hints on error message, which suggest this happens due to usage of more registers than the limit specified by the GPU for each…
fahad
  • 383
  • 1
  • 6
  • 16
2
votes
1 answer

python module compiled by SWIG returns wrong result

I'm just playing around SWIG to make python' module for specific C library. I got the trouble with double and float variables. Here an example: /***** simple.c *****/ #include double doublefun(double b){ printf("c(%g)",b); …
rth
  • 2,946
  • 1
  • 22
  • 27
2
votes
1 answer

Order of magnitude for double precision

What order of magnitude difference should I be expecting for a subtraction between two theoretically equal double precision numbers? I have two double precision arrays. They are expected to be theoretically same. They are both calculated by two…
Pradeep Kumar Jha
  • 877
  • 4
  • 15
  • 30
2
votes
1 answer

MATLAB vpa() doesn't compute variable-point number for expression with exponent?

Trying to use vpa() to compute a variable point number for a rational expression in an exponent: syms x; ans1 = x^(12345/67890) ans2 = vpa(x^(12345/67890),3) ans2_5 = vpa((12345/67890),3) ans3 = vpa(x*(12345/67890),3) The above shows the issue.…
Trevor
  • 181
  • 1
  • 7
2
votes
7 answers

How to determine whether a double variable has integer value?

I'm trying to find possible integer roots of a quadratic equation with Java. Here is a snippet from my code: double sqrtDiscriminant = Math.sqrt(b * b - 4 * a * c); double root1 = ((-1.0 * b) - sqrtDiscriminant) / (2.0 * a); double root2 =…
Juvanis
  • 25,802
  • 5
  • 69
  • 87
2
votes
2 answers

What is the optimum epsilon/dx value to use within the finite difference method?

double MyClass::dx = ?????; double MyClass::f(double x) { return 3.0*x*x*x - 2.0*x*x + x - 5.0; } double MyClass::fp(double x) // derivative of f(x), that is f'(x) { return (f(x + dx) - f(x)) / dx; } When using finite difference method…
hkBattousai
  • 10,583
  • 18
  • 76
  • 124
2
votes
1 answer

stringstream setprecision and floating point formatting

double value = 02369.000133699;//acutally stored as 2369.000133698999900 const std::uint32_t left = std::uint32_t(std::abs(value) < 1 ? 1: (1 + std::log10(std::abs(value)))); std::ostringstream out; out <<…
Guillaume Paris
  • 10,303
  • 14
  • 70
  • 145
1 2 3
…
15 16