Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
2
votes
1 answer

Adding 0 to a Float/Double Type in C++

I heard that floating point arithmetic like 0.1 + 0.2 may yield rounding error like 0.30000000000000004 due to binary floating point being used internally. But if I add a 0 to any floating point number in C++, does it guarantee to produce the same…
JavaMan
  • 4,954
  • 4
  • 41
  • 69
2
votes
3 answers

Why does std::cout print 4.9999999 as 5?

I was figuring out the difference between log(3) and log10(3), using this code: void testPrecisionError() { cout << log(243) / log(3) << " : " << int(log(243) / log(3)) << " : " << endl; cout << log10(243) / log10(3) << " :…
Shihao Xu
  • 1,160
  • 2
  • 10
  • 23
2
votes
2 answers

Use double as long as possible before casting to float?

I have a vector class (2D mathematical vector) with floats as its components. For some intermediate calculations I have the choice of using doubles. Should I use the intermediate double precision as long as possible, only casting to float at the…
jhyot
  • 3,733
  • 1
  • 27
  • 44
2
votes
1 answer

Float values UNION ALL query

I have two ms sql server tables with column of type float (not null). when I do union ALL in those two tables SELECT [float_column], Min(sourcename) AS ExistsInFile FROM (SELECT [float_column], 'File 1' AS SourceName …
Relativity
  • 6,690
  • 22
  • 78
  • 128
2
votes
1 answer

matlab double precision confusion

Matlab sets inputs to double precision by default, so if I input a=1/3 then the variable will be converted with double precision >> a=1/3 a =0.3333 >> whos('a') a 1x1 8 double However, when I input vpa(a,100)…
llxxee
  • 59
  • 1
  • 6
2
votes
2 answers

Understanding mpz_invert

im fairly new to Multiple Precision Arithmetic, and after a few days of trying to figure this out im at a loss. im trying to take the inverse of a number to a high number of deciaml places and have been attempting to work out how to do this using…
Why_Is_This_Hard
  • 99
  • 2
  • 5
2
votes
2 answers

%g gives Exponential Form of Value in ios

My Requirement There is one label having Price Value (Float Value) If it is -> Then It should be 1232.200 -> 1232.2 1232.243 -> 1232.243 1232.00 -> 1232 What I had Done till Now 1) Used %.2f lblPrice.text=[NSString…
Sneha
  • 880
  • 9
  • 24
2
votes
3 answers

Dividing uint64_t by numeric_limits::max() to a floating point representation

Given a uint64_t value, is it possible to divide it by std::numeric_limits::max() so that a floating point representation of the value results (0.0 to 1.0 representing 0 to 2^64-1)? Numbers bigger than max can be chalked up to undefined…
Stijn Frishert
  • 1,543
  • 1
  • 11
  • 32
2
votes
1 answer

Duplicated with a tiny magnitude difference

I'm using the duplicated function on a data.table object. It returns FALSE on two values that seem to be identical. Looking deeper into them, it appears that they have a tiny difference (of -1.867777e-14, but it can be any other near-zero…
Yonatan Lazar Telem
  • 369
  • 2
  • 13
2
votes
3 answers

Intel x86_64 assembly compare signed double precision floats

I've got a problem according to subject. In xmm0 register I have a value, e.g. -512.000000 And in xmm4: 0.000000. I try to compare the first value with zero and I cannot really achieve this. comisd xmm0, xmm4 COMISD instruction sets flags in a…
formateu
  • 157
  • 2
  • 17
2
votes
0 answers

Unintentional shortening (truncating) values in GraphicPath.AddLine

I'm loading coordinate values with GraphichsPath.AddLine command, then drawing them and I've noticed my map is not quite accurate. The code in question is as follows. I'm casting start and end coordinates from double to float since the command takes…
Romy
  • 407
  • 6
  • 25
2
votes
2 answers

Safe computation of Geometric Mean

I am looking for a reason to choose one of the following way to compute geometric mean of a long series of floating point x: take nth root of each x, then multiply all of them multiply all of them, then take nth root I have heard that for floating…
Patrick the Cat
  • 2,138
  • 1
  • 16
  • 33
2
votes
4 answers

What is the standard way to maintain accuracy when dealing with incredibly precise floating point calculations in C++?

I'm in the process of converting a program to C++ from Scilab (similar to Matlab) and I'm required to maintain the same level of precision that is kept by the previous code. Note: Although maintaining the same level of precision would be ideal. It's…
Paul Warnick
  • 903
  • 2
  • 13
  • 26
2
votes
1 answer

Precision of computing left eigenvectors with Matlab

Matlab defines a column left eigenvector w of the matrix A by the equation w*A=d w* where w* is the conjugate transpose of w. This implies that when you diagonalize the matrix A by the tranformation D=S^{-1}AS where D is a diagonal matrix and the…
Tarek
  • 1,060
  • 4
  • 17
  • 38
2
votes
1 answer

Intel x86_64 assembly, How to move between x87 and SSE2? (calculating arctangent of double)

I try to calculate arctangent of double precision float value, which I hold in xmm register. With normal float pointing it is possible to use old x87 instruction FPATAN, but how can I do this with double ?
formateu
  • 157
  • 2
  • 17
1 2 3
…
99 100