Anything related to double-precision floating-point arithmetic and data-types. Often used with reference to IEEE754 double-precision floating-point representation.
Questions tagged [double-precision]
240 questions
-1
votes
5 answers
Java Double object storing till two decimal places
Is it possible to store java.lang.Double to exactly two decimal places
Ex : 0.2 becomes 0.20
I modified the code to following
Double correlation = getSomeDoubleValue(label, key, date);
DecimalFormat df = new…
Lav
- 1,283
- 5
- 21
- 48
-1
votes
2 answers
Decimal Precision Lost after 15th Digit - Bad PI
Attempting to print more than 15 decimal digits of PI result in incorrect decimals printing after the 15th decimal. This despite the 30 correct decimal values being assigned and despite using long double to hold the value. The following test case…
David C. Rankin
- 81,885
- 6
- 58
- 85
-1
votes
2 answers
How to use Bitxor for Double Numbers?
I want to use xor for my double numbers in matlab,but bitxor is only working for int numbers. Is there a function that could convert double to int in Matlab?
saba 2085
- 21
- 1
- 4
-1
votes
1 answer
C++ log precision error for double vector array?
I have double vector array ( vector< double > ) and its max value ( 1 ) and min value ( 0 ).
When I did below sub operation, log gives (1.#INF00000000000000000) for max value's index that ( 1 - 1.000000000000001 ) ( double precison ).
How can I…
user2227305
- 11
- 3
-1
votes
2 answers
Using SetPrecision incorrectly
With the following 2 functions focus on the rows with the * comments. When the cout<< statement executes, there is no decimal place showing. 3021680380 /10000000 turns into 302. instead of 302.1680.
void convert(){
setprecision(4); …
theB3RV
- 894
- 4
- 13
- 33
-1
votes
4 answers
Assigning 2^31 to a float or double
I have to multiply an integer value by 2^31. I have googled it, and it looks like doubles have a range among 2.23e-308 <= |X| <= 1.79e308 when using 64 bits, and a float among 1.18e-38 <= |X| <= 3.40e38.
That's a lot more than what I need. But it…
Roman Rdgz
- 12,836
- 41
- 131
- 207
-2
votes
1 answer
Double printing same as float
Why is my double variable storing and printing as a float. Using Xcode.
#import
#import
long double doubleTest = .123456789101112;
float floatTest = .123456789101112;
NSLog(@"Float %f vs Double %Lf",…
Peter
- 1,053
- 13
- 29
-2
votes
1 answer
controlling overflow and loss in precision while multiplying doubles
ques:
I have a large number of floating point numbers (~10,000 numbers) , each having 6 digits after decimal. Now, the multiplication of all these numbers would yield about 60,000 digits. But the double range is for 15 digits only. The output…
bnks452
- 1
- 1
-2
votes
1 answer
Excel Spreadsheet not giving same answer as C# program
I have trouble with the following code. It is not giving the same answer in excel as C# and I'm sure the excel answer is correct. I have already tried to change integer values to double by adding a decimal point. Please advise.
the excel…
Alfred Waligo
- 2,809
- 3
- 18
- 26
-2
votes
1 answer
Precision of calculations
I am doing a calculation in Fortran on a double-precision variable, and after the calculation the variable gets the value -7.217301636365630e-24.
However, when I do the same computation in Matlab, the variable just gets the value 0. Is there a way…
BillyJean
- 1,537
- 1
- 22
- 39
-2
votes
5 answers
How to always round in two decimals
For a project at school, I have to make a java program to retrieve data from a database.
This is my code:
import java.text.*;
import java.util.*;
public class Tijdstip
{
public Tijdstip()
{
}
public double testTijd(String tijdstip1)
…
Denny
- 1,766
- 3
- 17
- 37
-2
votes
1 answer
how can save a double type correctly in grails?
basically the domain to save not respect the floating point, as it makes it to another format
i have the next domain
class Location {
double latitude
double longitude
static belongsTo=[ product: Product ]
static constraints = { …
tec-net
- 26
- 5
-2
votes
2 answers
Visual Express 2010 double variable precision
I have just started out coding in c++ but have quite a bit of previous experience with MATLAB and MySql. I am trying to calculation some compounding numbers hence accuracy is key. I have tried to do this using double numbers but for some reason I…
Mkoll
- 1
- 3
-3
votes
1 answer
FORTRAN - Correct values are calculated in simple precision, but erroneous solutions in double precision?
I have the following code, regarding Newton's method for calculating zeros in multivariable vectorial functions:
program Newton_Method_R_N_R_N_1_Numeric
use Inverse_Matrices
implicit none
integer :: i, j !INDEX
real…
Steve
- 109
- 4
-4
votes
3 answers
c++: why 0 is displayed as 1.29571e+261
I am trying to create my first class in c++. I am making file called geometryitems.cpp and doing this:
using namespace std;
class Point
{
double x, y, z;
public:
// constructor
Point(double x, double y, double z)
{
}
…
user1329187