Stack Overflow
Stack Overflow

Questions tagged [double-precision]

Anything related to double-precision floating-point arithmetic and data-types. Often used with reference to IEEE754 double-precision floating-point representation.

240 questions
5
votes
6 answers

C# High double precision

I'm writing a function that calculates the value of PI, and returns it as a double. So far so good. But once the function gets to 14 digits after the decimal place, it can't hold any more. I'm assuming this is because of the double's limited…
Entity
  • 7,972
  • 21
  • 79
  • 122
5
votes
6 answers

Best base type to deal with linear algebra

I'm writing a small and inadequate linear algebra library in C++ for a project (I'm sorry). I'm implementing matrices and operations using double precision numbers. I'm doing right? Should I implement a template class instead? Is there a more…
tunnuz
  • 23,338
  • 31
  • 90
  • 128
5
votes
2 answers

Does Fortran have inherent limitations on numerical accuracy compared to other languages?

While working on a simple programming exercise, I produced a while loop (DO loop in Fortran) that was meant to exit when a real variable had reached a precise value. I noticed that due to the precision being used, the equality was never met and the…
EMiller
  • 2,792
  • 4
  • 34
  • 55
5
votes
1 answer

Confusion about kinds in FORTRAN

I have been in the process of writing a FORTRAN code for numerical simulations of an applied physics problem for more than two years and I've tried to follow the conventions described in Fortran Best Practices. More specifically, I defined a…
Toon
  • 187
  • 11
5
votes
2 answers

Is integer multiplication implemented using double precision floating point exact up until 2^53?

I ask because I am computing matrix multiplications where all the matrix values are integers. I'd like to use LAPACK so that I get fast code that is correct. Will two large integers (whose product is less than 2^53), stored as doubles, when…
Steven Lu
  • 41,389
  • 58
  • 210
  • 364
4
votes
1 answer

Output precision is higher than double precision

I am printing some data from a C++ program to be processed/visualized by ParaView, but I am having a problem with floating point numbers. Paraview supports both Float32 and Float64 data types. Float64 is equivalent to double with the typical limits …
iluvatar
  • 872
  • 10
  • 21
4
votes
1 answer

C++ writing and reading matrices of doubles from a binary file

Here comes a new issue after my previous question : I've extended the code to perform matrix binary files I/O and when testing a simple write and read operation, I retrieved only the first line of the matrix... I don't have managed to find my error,…
Andy
  • 429
  • 1
  • 6
  • 17
4
votes
3 answers

C++ writing and reading doubles from a binary file

I want to perform disk I/O operations for a program that takes too much RAM. I use matrices of doubles and think writing them to disk as bytes is the fastest way (I need to preserve the double precision). How to do it with portability? I found this…
Andy
  • 429
  • 1
  • 6
  • 17
4
votes
2 answers

What does the "double" do in ceil(double)?

I have a number (let's say, 34), and I want to find its next multiple of ten. I can do this by: Dividing the number by 10 Rounding it up to a whole number Multiplying by 10. After a bit of research, I discovered that this is the code for that in…
Ric Levy
  • 966
  • 1
  • 15
  • 33
4
votes
2 answers

MATLAB: Converting a uint32 (4-byte) value to the corresponding IEEE single-precision floating-point form

In MATLAB (r2009b) I have a uint32 variable containing the value 2147484101. This number (its 4-bytes) has been extracted from a digital machine-vision camera in a grabbing process. According to what I understand it holds the single-precision form…
Ole Thomsen Buus
  • 1,333
  • 1
  • 9
  • 24
4
votes
1 answer

NSDate and double precision problem

Here is the code NSDate* d = [NSDate dateWithTimeIntervalSince1970:32.4560]; double ti = [d timeIntervalSince1970]; NSLog(@"Interval: %f %f %f %f",ti,32.4560,ti*1000.0,32.4560*1000.0); the output is Interval: 32.456000 32.456000 32455.999970…
teerapap
  • 5,303
  • 7
  • 33
  • 40
4
votes
2 answers

Can std::uniform_real_distribution(0,1) return a value greater than 0.99999999999999994?

From the C++11 header , I was wondering if a std::uniform_real_distribution object can spit out a double that's greater than 0.99999999999999994? If so, multiplying this value by 2 would equal 2. Example: std::default_random_engine…
starpax
  • 130
  • 5
4
votes
4 answers

Java DecimalFormat losing precision while formatting double

When i execute the below code: public class Test { public static void main(String args[]){ DecimalFormat format = new DecimalFormat(); Double value = new Double(-1350825904190559999913623552.00); StringBuffer buffer =…
Anusha Pachunuri
  • 1,389
  • 4
  • 18
  • 39
4
votes
1 answer

Unexpected error in Julia set rendering

I am playing with Mandelbrot and Julia sets and I encountered interesting problem. The Mandelbrot set can be rendered in double precision until zooms of around 2^56 at any place. However, the Julia set sometimes produces artifacts much sooner like…
NightElfik
  • 4,328
  • 5
  • 27
  • 34
4
votes
1 answer

Why is JavaScript's number *display* for large numbers inaccurate?

So in JavaScript, 111111111111111111111 == 111111111111111110000. Just type any long number – at least about 17 digits – to see it in action ;-) That is because JavaScript uses double-precision floating-point numbers, and certain very long numeric…
purefanatic
  • 933
  • 2
  • 8
  • 23
1 2
3
…
15 16