Anything related to double-precision floating-point arithmetic and data-types. Often used with reference to IEEE754 double-precision floating-point representation.
Questions tagged [double-precision]
240 questions
1
vote
1 answer
How are double-precision floating-point numbers converted to single-precision floating-point format?
Converting numbers from double-precision floating-point format to single-precision floating-point format results in loss of precision. What's the algorithm used to achieve this conversion?
Are numbers greater than 3.4028234e+38 or lesser than…
Aadit M Shah
- 72,912
- 30
- 168
- 299
1
vote
2 answers
CURAND_STATUS_DOUBLE_PRECISION_REQUIRED is undefined
While building my CUDA project I get the following error:
cutil_inline_runtime.h(328): error: identifier "CURAND_STATUS_DOUBLE_PRECISION_REQUIRED" is undefined
So I started googling. Since I couldn't find the solution (nor did I find the actual…
zkristic
- 629
- 1
- 9
- 24
1
vote
1 answer
Why won't MySQL get the correct data?
I'm using CodeIgniter's Active Record and my code is:
$current_balance = $this->get_campaign_balance($click_report['campaign_id']);
$campaign_ledger = array(
'campaign_id' => $click_report['campaign_id'],
'description' => "Click…
Shamoon
- 41,293
- 91
- 306
- 570
0
votes
1 answer
Glib: g_ascii_dtostr imprecise?
I am a little bit intrigued by the way some glib functions such as "g_ascii_dtostr" (and the GKeyFile functions using doubles) work.
Consider this line:
gchar buf[30];
g_message("Double: %f, as String: %s", 0.2, g_ascii_dtostr(buf, 30, 0.2));
Which…
Ancurio
- 1,698
- 1
- 17
- 32
0
votes
2 answers
Precision not working or devcpp failure in long double
I have used this :
long double f =79228162514264337593543950336.0;//maximum ; 2 ^ 96 because f is 12 bytes
cout.precision(30);
cout<
S.A.Parkhid
- 2,772
- 6
- 28
- 58
0
votes
2 answers
Floating point error in representation?
when i make this multiplication
0.94 * 8700
the output is
8177.999999999999
but it should have been
8178
i'm using java , but i don't think this error is related to a particular Programming language
now my question is ... why this happened…
Ahmad Ibrahem
- 97
- 1
- 6
0
votes
2 answers
Converting/expressing double number in non-exponent/short form in Javascript
I have a double in Javascript whose value is, for example, 1.0883076389305e-311.
I want to express it in the following form, using as example the 'bc' utility to calculate the expanded/higher precision/scale form:
$…
user656208
0
votes
1 answer
Mixed precision assignment
I am working with a large Fortran code that uses XLF compiler for Power7
It uses these flags to set the precision to 64bit: -qrealsize=8 -qintsize=8
However a recently imported piece of code has lines similar to this:
real :: var1d(nvol)
real ::…
0
votes
1 answer
Altering precision in HIVE
I want to set the precision of a column in my hive table to have a precision of 11 (11 places after the decimal). So say I have the double var:
var = 215.6666666666667
I want to get this down to two less precision:
var = 215.66666666667
Is there a…
Danzo
- 553
- 3
- 13
- 26
0
votes
1 answer
Why precision problems in a floating point number varies according to values?
I was wondering why precision problems in floating point numbers are different with different values:
#include
#include
int main ()
{
std::cout << std::setprecision(20);
double d1(1.0);
std::cout << d1 <<…
user5117637
0
votes
4 answers
Is float better than double sometimes?
I was solving this problem on spoj http://www.spoj.com/problems/ATOMS/. I had to give the integral part of log(m / n) / log(k) as output. I had taken m, n, k as long long. When I was calculating it using long doubles, I was getting a wrong answer,…
In78
- 440
- 4
- 17
0
votes
1 answer
How to obtain Fortran precision in MatLAB
I have a piece of code written in Fortran and in Matlab. They do exactly the same calculation, namely
Construct a tanh -field and find its Laplacian
Multiply some terms together
The result of this multiplication yields a matrix, whose (4,4)th and…
BillyJean
- 1,537
- 1
- 22
- 39
0
votes
0 answers
Mips single to double floating point percision
I have this program in mips and I wasn't to change it to double precision. It looks like single and double precision floating instructions have the same instructions but instead of .s it is .d If anyone has commens or help it would help my out a…
jmurphy1267
- 65
- 1
- 9
0
votes
1 answer
Why I can not calculate double value?
I want to calculate the value 0.95. Here is my method:
public static final int VAR = 5;
private static double getDouble(){
double dis = (double)(VAR/100);
dis = (double)(1-dis);
return dis;
}
However, it output 1.0?? If I…
sammy333
- 1,384
- 6
- 21
- 39
0
votes
2 answers
Converting a number to normalized scientific notation
I'm trying to create a method converting a number to normalized scientific notation, here is the code that I'm using to calculate mantissa and exponent:
ConvNSN 1000, M, P
MsgBox M & "e" & P
Sub ConvNSN(N, M, P)
If N = 0 Then
P = 0
…
omegastripes
- 12,351
- 4
- 45
- 96