Stack Overflow
Stack Overflow

Questions tagged [double-precision]

Anything related to double-precision floating-point arithmetic and data-types. Often used with reference to IEEE754 double-precision floating-point representation.

240 questions
2
votes
0 answers

looking for inversed double values

I am looking for inverse double values if( temp1.p[j] == -temp2.p[j] ) { cout << "Inverse Value \n"; } Problem I have is that when I multiple the following value they are not always completely exact for instance the last 3 digits. temp2.p[j] = …
codem
  • 283
  • 1
  • 2
  • 9
2
votes
2 answers

iOS: Limit precision of datatype "double" without converting to NSString

Here I am having a query about how double works with their precision. I created a sample where in I entered the double values as below: double d = 2.0126161; double d1 = 2.0126162; double d2 = 2.0126163; double d3 = 2.0126164; double d4 =…
Parth Bhatt
  • 19,381
  • 28
  • 133
  • 216
2
votes
2 answers

What methods are there for finding the element of an array containing a specific real value?

In a Fortran code, I have an array a with values ranging uniformly from 0.0 to 1.2 and I am looking for the index of the value which is closest to 1.0. There don't seem to be any intrinsics to do this in Fortran, so I made a hacky workaround with…
Makkasu
  • 166
  • 1
  • 1
  • 10
2
votes
0 answers

Javascript number conversion

why the conversion fail? var ccc=10206229832728593; alert("saveme.php?id="+ccc.toString() ) ; var ccc=10152714408613803; alert("saveme.php?id="+ccc.toString() ) ; the first var is decreased by 1 and the second one is increased by 1. I'm going…
mr_gian55
  • 43
  • 8
2
votes
1 answer

double precision error when converting to scientific notation

I'm building a program to to convert double values in to scientific value format(mantissa, exponent). Then I noticed the below 369.7900000000000 -> 3.6978999999999997428 68600000 -> 6.8599999999999994316 I noticed the same pattern for several…
vibz
  • 157
  • 1
  • 12
2
votes
2 answers

How to use fmod and avoid precision issues

I'm going to boil this problem down to the simplest form: Let's iterate from [0 .. 5.0] with a step of 0.05 and print out 'X' for every 0.25 multiplier. for(double d=0.0; d<=5.0; d+=0.05) { if(fmod(d,0.25) is equal 0) print 'X'; } This…
c0dehunter
  • 6,412
  • 16
  • 77
  • 139
2
votes
2 answers

SQL Error - missing keyword

Whats wrong with this query : I am getting following error SQL Error: ORA-00905: missing keyword 00905. 00000 - "missing keyword" it says error at 4th row. Please advise CREATE TABLE ORDERS ( ID INT NOT NULL, ord_date DATE, AMOUNT …
coder1608
  • 171
  • 6
  • 14
2
votes
1 answer

What is the precision of std::erf?

C++11 introduced very useful math functions in the standard like erf and erfc. There are mentions about "guaranteed underflow" for inputs greater or smaller than certain values, but I don't know enough about floating point representation to…
Jonathan H
  • 7,591
  • 5
  • 47
  • 80
2
votes
2 answers

rounding error in C++ advice

I am trying to model a dynamical system in C++ and have some issues with the precision of stored values. I have read the question on the floating point precision, but am unable to figure out what is going wrong in my case. So, the issue is about…
miha priimek
  • 919
  • 6
  • 20
2
votes
2 answers

Math.pow(10,6) = 999999.999999 on Doogee DG800 Android phone

I know that Math.pow(10,6) performs the computation using doubles which means the precision is not absolute. The current release of Google Authenticator uses following code to compute codes: int codeLength = 6; .... int code = truncatedHash % (int)…
LRA
  • 462
  • 3
  • 8
2
votes
2 answers

Incorrect values for double calculation

I am doing a simple calculation in VB.Net, following is the code. Dim total As String = "192.04" Dim paid As String = "200" Dim change As String = "7.96" '//This prints -7.99360577730113E-15 (Incorrect) MsgBox((CDbl(total) - CDbl(paid)) +…
codeGEN
  • 686
  • 2
  • 18
  • 45
2
votes
2 answers

Generating random integer between 1 and infinity

I would like to create an integer value between 1 and infinity. I want to have a probability distribution where the smaller the number is, the higher the chance it is generated. I generate a random value R between 0 and 2. Take the series I want to…
PEC
  • 593
  • 1
  • 5
  • 16
2
votes
2 answers

Issue with java BigDecimal storing garbage value

BigDecimal b = new BigDecimal(0.05); System.out.println(b); Output: 0.05000000000000000277555756156289135105907917022705078125 How to handle this ?
Vikash Singh
  • 13,213
  • 8
  • 40
  • 70
2
votes
2 answers

g++ double precision arm vs intel

Here is an image showing an output of the same program in intel and then in ARM: http://screencast.com/t/1eA64D4rF Both show output from reading a binary file with the numbers in the first column being of double precision floating point format. Why…
valueRunner
  • 23
  • 3
2
votes
1 answer

Why does the precision of a double differ between C and C#?

I'm trying to rewrite part of an old system as a C# program. The old programs where written in C. Both programs read blob files into a byte array and fill an object/struct with the data. In the original C code this is done with…
user3070194
  • 33
  • 5
1 2 3
…
15 16