Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
16
votes
8 answers

C# Convert object to Decimal

I'm trying to convert an object with the value 0.39999999999999997 to a decimal variable without losing the precision. object d = 0.39999999999999997; I've tried the following methods. decimal val1 = Convert.ToDecimal(d); // val1 = 0.4 object val2…
Chathura W
  • 351
  • 1
  • 4
  • 14
16
votes
3 answers

C# Unable to cast object of type 'System.Double' to type 'System.Single'

before judging that this question is already answered, please read the description. I have this simple code below: Dictionary d = new Dictionary(); d.Add("key" , 30d); System.Diagnostics.Debug.WriteLine($" TYPE OF…
With A SpiRIT
  • 478
  • 3
  • 6
  • 17
16
votes
7 answers

Are arithmetic operations on double variables holding integer values exact?

Let's say I've got two integer values stored in double variables, e. g.: double x = 100.0; double y = 7.0; May I safely assume that any arithmetic operation on these two double variables that would yield an integer result, will return an exact…
MicSim
  • 26,265
  • 16
  • 90
  • 133
16
votes
1 answer

Emulating fixed precision in python

For a university course in numerical analysis we are transitioning from Maple to a combination of Numpy and Sympy for various illustrations of the course material. This is because the students already learn Python the semester before. One of the…
JKaerts
  • 173
  • 4
16
votes
3 answers

Difference between R's sum() and Armadillo's accu()

There are small differences in the results of R's sum() function and RcppArmadillo's accu() function when given the same input. For example, the following code: R: vec <- runif(100, 0, 0.00001) accu(vec) sum(vec) C++: //…
Matthew Lueder
  • 953
  • 2
  • 12
  • 25
16
votes
2 answers

Extreme numerical values in floating-point precision in R

Can somebody please explain me the following output. I know that it has something to do with floating point precision, but the order of magnitue (difference 1e308) surprises me. 0: high precision > 1e-324==0 [1] TRUE > 1e-323==0 [1] FALSE 1: very…
user3370602
16
votes
3 answers

How do I retain precision for a Fortran MPI program in a portable way?

I have a Fortran program where I specify the kind of the numeric data types in an attempt to retain a minimum level of precision, regardless of what compiler is used to build the program. For example: integer, parameter :: rsp =…
rks171
  • 1,000
  • 11
  • 26
16
votes
1 answer

Are there any whole numbers which the double cannot represent within the MIN/MAX range of a double?

I realize that whenever one is dealing with IEEE 754 doubles and floats, some numbers can't be represented especially when one tries to represent numbers with lots of digits after the decimal point. This is well understood but I was curious if…
Brett
  • 4,066
  • 8
  • 36
  • 50
16
votes
1 answer

Numpy: Difference between dot(a,b) and (a*b).sum()

For 1-D numpy arrays, this two expressions should yield the same result (theorically): (a*b).sum()/a.sum() dot(a, b)/a.sum() The latter uses dot() and is faster. But which one is more accurate? Why? Some context follows. I wanted to compute the…
user2304916
  • 7,882
  • 5
  • 39
  • 53
16
votes
1 answer

SQL Server decimal scale length - can be or has to be?

I have really simply question about DECIMAL (and maybe NUMERIC) type in SQL Server 2008 R2. MSDN said: (scale) The maximum number of decimal digits that can be stored to the right of the decimal point. Scale must be a value from 0 through p. I…
Jan Drozen
  • 894
  • 2
  • 12
  • 28
16
votes
1 answer

Point in Polygon algorithm giving wrong results sometimes

I saw on StackOverflow a "point in polygon" raytracing algorithm that I implemented in my PHP Code. Most of the time, it works well, but in some complicated cases, with complex polygons and vicious points, it fails and it says that point in not in…
user1527491
  • 895
  • 10
  • 22
16
votes
3 answers

What is c printf %f default precision?

I'm curious: If you do a printf("%f", number); what is the precision of the statement? I.e. How many decimal places will show up? Is this compiler dependent?
spbots
  • 1,513
  • 1
  • 15
  • 22
15
votes
3 answers

Precision of floats with printf

As everybody knows, you have limited precision when you use printf to output the value of a float. However, there is a trick to increase the accuracy in the output, as this example shows: #include int main() { float f = 1318926965; …
Mr Lister
  • 45,515
  • 15
  • 108
  • 150
15
votes
5 answers

How to set precision of a float

Can someone explain me how to choose the precision of a float with a C function? Examples: theFatFunction(0.666666666, 3) returns 0.667 theFatFunction(0.111111111, 3) returns 0.111
Zat42
  • 2,471
  • 4
  • 22
  • 36
15
votes
4 answers

Determine MAX Decimal Scale Used on a Column

In MS SQL, I need a approach to determine the largest scale being used by the rows for a certain decimal column. For example Col1 Decimal(19,8) has a scale of 8, but I need to know if all 8 are actually being used, or if only 5, 6, or 7 are being…
Bo Flexson
  • 481
  • 1
  • 6
  • 9
1 2 3
…
99 100