Stack Overflow
Stack Overflow

Questions tagged [double-precision]

Anything related to double-precision floating-point arithmetic and data-types. Often used with reference to IEEE754 double-precision floating-point representation.

240 questions
1
vote
2 answers

For-loop with double in Java

Problem. I have a String of double "0.4 0.3 2.1 etc". I need to loop in this list and count occurencies of each number. I used StringTokenizer stokens = new StringTokenizer(values); while(stokens.hasMoreTokens()) { Double tempKey = new…
Tizianoreica
  • 2,142
  • 3
  • 28
  • 43
1
vote
2 answers

Understanding DBL_MAX

I just read about the IEEE 754 standard in order to understand how single-precision and double-precision floating points are implemented. So I wrote this to check my understanding: #include #include int main() { double foo =…
aldeb
  • 6,588
  • 5
  • 25
  • 48
1
vote
2 answers

DISLIN double-precision libraries

I am trying to compile a program written in FORTRAN that plots graphs using the DISLIN libraries, but all data is in double precision. I cannot lose this precision, so passing everything to simple precision is not an option. When I attempt to link…
Steve
  • 41
  • 6
1
vote
3 answers

Double-precision error using Dislin

I get the following error when trying to compile: call qplot (Z, B, m + 1) 1 Error: Type mismatch in argument 'x' at (1); passed REAL(8) to REAL(4) Everything seems to be in double precision so I can't help but think it is a Dislin…
Steve
  • 41
  • 6
1
vote
1 answer

Are there any built-in .NET alternatives for using power of a number (Math.Pow) without errors?

Math.Pow seems to be not working correctly for big results. Probably that is because it uses double for calculations (How is Math.Pow() implemented in .NET Framework?). For example: public static void Main() { …
Nikolay Kostov
  • 16,433
  • 23
  • 85
  • 123
1
vote
1 answer

trim values of type Exponential type which are retrieved from ResultSet .getDouble() in java

I get my values into list from Resultset.getDouble().The values retrieved are of exponential type such as 9.191152283705151E-10 and some values are 89.98999786376953 type. I want exponential values to be trimmed as 9.191E-10 and other values up to…
tiddi rastogi
  • 526
  • 3
  • 10
  • 33
1
vote
1 answer

Fortran Floating Error while performing summation within DO loop?

I have four double precision real numbers (n1, n2, n3, & n4) in an array (n). The weird thing that when I calculate the sum of these four numbers within a DO loop and then calculate the sum directly I don't get the same exact number! Note that I am…
user3203131
  • 75
  • 1
  • 1
  • 4
1
vote
3 answers

Precision loss with java.lang.Double

Say I have 2 double values. One of them is very large and one of them is very small. double x = 99....9; // I don't know the possible max and min values, double y = 0,00..1; // so just assume these values are near max and min. If I add those…
Utku Ufuk
  • 318
  • 1
  • 10
  • 24
1
vote
2 answers

estimate the size of a binary file

I am very confused about this. I dump out a 3D array of size 16000*4*2, all elements of which are DOUBLE PRECISION, and I thought I should have gotten a file of size: 16000*4*2*8 bytes/dp = 1,024,000 bytes. But I keep getting 2,048,000 bytes. And I…
user2008151314
  • 680
  • 6
  • 10
1
vote
1 answer

Oracle 11g -> Oracle Client -> Pro*C Program Double Precision representation

A NUMBER(12,3) field in an 11g R2 Oracle db instance stores the number 271.448 It is FETCHed from a cursor into a double variable within a 32 bit Pro*C (version 11.1.0.6). preprocessed C program with Visual Studio 2010 being used for compilation of…
Stephen Emm
  • 83
  • 1
  • 1
  • 5
1
vote
3 answers

Most effective Algorithm to find maximum of double-precision values

What is the most effective way of finding a maximum value in a set of variables? I have seen solutions, such as private double findMax(double... vals) { double max = Double.NEGATIVE_INFINITY; for (double d : vals) { if (d > max) max = d; } …
MarkusWillson
  • 411
  • 1
  • 3
  • 11
1
vote
1 answer

what is the correct way to calculate the "double's range"?

In 64 bits platform, the double's range are following: -2.22507e-308 ~ +2.22507e-308 -1.79769e+308 ~ +1.79769e+308 and I followed the IEEE754 standard and tried to calculate the double's range by following equation, x = (-1)^s * (1+fraction) *…
RyanLiu
  • 1,557
  • 2
  • 18
  • 27
1
vote
9 answers

Floating point arithmetic is too reliable

I understand that floating point arithmetic as performed in modern computer systems is not always consistent with real arithmetic. I am trying to contrive a small C# program to demonstrate this. eg: static void Main(string[] args) { …
mcoolbeth
  • 467
  • 1
  • 6
  • 19
1
vote
1 answer

PostgreSQL: Modifying column data type from Character Varying to Double Precision

I am encountering some problems regarding the alteration of column data types -- specifically from character varying to double precision. The goal of this modification is so I can perform SUM and average on this column. Another way I've been trying…
asadasan
  • 13
  • 1
  • 3
1
vote
2 answers

Double comparison (again)

I have a "golden source" algorithm that is completely accurate, i.e. the output is accurate to double precision. I want to compare another algorithm and determine whether it is also accurate to double precision. For example, the following three…
Rai
  • 1,328
  • 11
  • 20
1 2 3
…
15 16