Anything related to double-precision floating-point arithmetic and data-types. Often used with reference to IEEE754 double-precision floating-point representation.
Questions tagged [double-precision]
240 questions
0
votes
2 answers
Is there a function in c++ using opencv which allows to convert an image to double precision as im2double in MATLAB?
I'm translating some code developed in MATLAB to C++ using OpenCV. I'm loading an image in the RGB format like:
Mat img;
img = imread('path to directory', CV_LOAD_IMAGE_COLOR);
I just want to convert this image to double precision as im2double()…
user3455534
- 1
- 3
0
votes
3 answers
Possible loss of precision double
int leccs1 = q1+q2+r1+swhw1;
System.out.println("LecCS = "+ leccs1+ "/60");
int lecprelim = lcp+lcpo;
System.out.println("LecPrelim = "+ lecprelim+ "/100");
double x = 0.50;
int lecgrade = ((x * leccs1) + (x *…
0
votes
1 answer
Data type double truncating precision
I have values stored as 0.05 and 0.10 in the database. I am creating a list using c# code to get these values in the list using :
foreach (var objST in obj.LstTerms)
{
SubcriptionTermsModel objSubcriptionTerms = new SubcriptionTermsModel();
…
user3346478
- 37
- 3
0
votes
1 answer
Numerical precision of implementation for convex quadrilateral area
I have implemented a method to compute a convex quadrilateral area in R3. The method works fine, but I am having numerical precision problems in the 8th decimal place. Take a look on the method:
internal static double GetTriangleArea(double ax,…
TapiocaCom
- 353
- 5
- 16
0
votes
1 answer
what is the precision difference between a double and a very long integer?
Take
36103009879073133562313702394913733
36103009879073133562313702394913733.0
as an example,
What is the difference in precision when representing a very long integer into double in C++?
LCFactorization
- 1,652
- 1
- 25
- 35
0
votes
2 answers
Java round() with double
im trying to program a little java program, that finds null points in a given function f(x).
This is my idea:
I have X1 and X2 (User input) to define an area where to search for the null point. For example -5 and 5.
private static double…
0
votes
1 answer
Error: Possible Loss Of Precision
This is my code:
static int compoundBalance(double db, double dbTwo, double dbThree) {
if(dbThree == 0) return db;
return (1 + dbTwo)*compoundBalance(db, dbTwo, dbThree-1);
}
And I get these two errors. I'm not sure what to make of them. Any…
user1378762
- 53
- 1
- 9
0
votes
1 answer
Is it useful to have intermediate results in doubles, when input and output are floats?
I am wondering about the following: I have a method that takes a float as input and returns a float after some calculation. The calculations may involve anything like addition, division, logarithms, powers etcetera. While doing so, the intermediate…
Yellow
- 3,955
- 6
- 45
- 74
0
votes
1 answer
Double Precision variations in OpenCL printf
I have a daft problem that I'd like someone to explain to me please. I have a simple OpenCL kernel that just takes in a double, prints it out inside the kernel, and then copies it back to the host. I've noticed that when I printf on the device…
SciCobalt
- 3
- 1
0
votes
3 answers
Large double mathematical multiplication in C#
I want to multiply this number:
5374711027510012111075768211110475111691021051041057653548210911210211112250867
66690120741165250567278571217510410482757487
with this…
Rila Nera
- 49
- 7
0
votes
2 answers
Why does exp(log(6)) == exp(log(3)) * 2 return FALSE in R?
In R, why is this false for y > 2?
y <- c(1, 2, 3, 4, 5)
x <- 2*y
exp(log(x)) == exp(log(y)) * 2
[1] TRUE TRUE FALSE FALSE FALSE
user1322720
0
votes
1 answer
Is it possible to cast NaN from "NaN" (string) ( using toFixed() )
So I got why to fixed returns a String, with this in mind is it possible to cast the "NaN" string into NaN to keep the following expression workable?
var numberWithPrecission = (1*"A").ToFixed(2) || 0;
Jonathan DS
- 2,050
- 5
- 25
- 48
0
votes
1 answer
Data type mismatch in fortran
I've written a rudimentary algorithm in Fortran 95 to calculate the gradient of a function (an example of which is prescribed in the code) using central differences augmented with a procedure known as Richardson extrapolation.
function f(n,x)
!…
pr0gramR
- 126
- 1
- 2
- 12
0
votes
1 answer
Matlab: Loss of precision in calculations. Scaling of variables possible?
Today I ran into a problem with precision in Matlab:
Tp = a./(3600*sqrt(g)*sqrt(K).*u.*Sd*sqrt(bB))
where
a =
346751.503002533
g =
9.81
bB =
2000
Sd =
749.158805838953
…
TheodorBecker
- 249
- 1
- 17
0
votes
2 answers
Accuracy of Matlab linspace and range versus Kahan summation algorithm
I'm using the Matlab linspace function and the range : operator to obtain equally spaced vectors, but I'm unespectedly receiving unequally spaced numbers. My code is the following:
format long
x1 = linspace(3,5,20);
diff(x1)
x2 =…
Vitality
- 20,705
- 4
- 108
- 146