Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
83
votes
2 answers

Why is 199.96 - 0 = 200 in SQL?

I have some clients getting weird bills. I was able to isolate the core problem: SELECT 199.96 - (0.0 * FLOOR(CAST(1.0 AS DECIMAL(19, 4)) * CAST(199.96 AS DECIMAL(19, 4)))) -- 200 what the? SELECT 199.96 - (0.0 * FLOOR(1.0 * CAST(199.96 AS…
Silverdust
  • 1,503
  • 14
  • 26
81
votes
7 answers

Is there any way to get current time in nanoseconds using JavaScript?

So, I know I can get current time in milliseconds using JavaScript. But, is it possible to get the current time in nanoseconds instead?
Carlos Melo
  • 3,052
  • 3
  • 37
  • 45
73
votes
2 answers

In MATLAB, are variables REALLY double-precision by default?

This question arose out of something strange that I noticed after investigating this question further... I always understood MATLAB variables to be double-precision by default. So, if I were to do something like declare a variable with 20 digits…
gnovice
  • 125,304
  • 15
  • 256
  • 359
71
votes
6 answers

Why is 24.0000 not equal to 24.0000 in MATLAB?

I am writing a program where I need to delete duplicate points stored in a matrix. The problem is that when it comes to check whether those points are in the matrix, MATLAB can't recognize them in the matrix although they exist. In the following…
Kamran Bigdely
  • 7,946
  • 18
  • 66
  • 86
71
votes
7 answers

Adjusting decimal precision, .net

These lines in C# decimal a = 2m; decimal b = 2.0m; decimal c = 2.00000000m; decimal d = 2.000000000000000000000000000m; Console.WriteLine(a); Console.WriteLine(b); Console.WriteLine(c); Console.WriteLine(d); Generates this…
Amy B
  • 108,202
  • 21
  • 135
  • 185
70
votes
5 answers

Are all integer values perfectly represented as doubles?

My question is whether all integer values are guaranteed to have a perfect double representation. Consider the following code sample that prints "Same": // Example program #include #include int main() { int a = 3; int b =…
Thomas
  • 4,696
  • 5
  • 36
  • 71
68
votes
8 answers

Emulate "double" using 2 "float"s

I am writing a program for an embedded hardware that only supports 32-bit single-precision floating-point arithmetic. The algorithm I am implementing, however, requires a 64-bit double-precision addition and comparison. I am trying to emulate double…
user210870
65
votes
4 answers

Convert Java Number to BigDecimal : best way

I am looking for the best way to convert a Number to a BigDecimal. Is this good enough? Number number; BigDecimal big = new BigDecimal(number.toString()); Can we lose precision with the toString() method ?
user1721413
63
votes
2 answers

Why does double in C print fewer decimal digits than C++?

I have this code in C where I've declared 0.1 as double. #include int main() { double a = 0.1; printf("a is %0.56f\n", a); return 0; } This is what it prints, a is…
Raghavendra Gujar
  • 471
  • 4
  • 6
63
votes
5 answers

Set the display precision of a float in Ruby

Is it possible to set the display precision of a float in Ruby? Something like: z = 1/3 z.to_s #=> 0.33333333333333 z.to_s(3) #=> 0.333 z.to_s(5) #=> 0.33333 Or do I have to override the to_s method of Float?
salt.racer
  • 21,903
  • 14
  • 44
  • 51
59
votes
7 answers

Double precision - decimal places

From what I have read, a value of data type double has an approximate precision of 15 decimal places. However, when I use a number whose decimal representation repeats, such as 1.0/7.0, I find that the variable holds the value of…
nf313743
  • 4,129
  • 8
  • 48
  • 63
56
votes
10 answers

Are doubles faster than floats in C#?

I'm writing an application which reads large arrays of floats and performs some simple operations with them. I'm using floats, because I thought it'd be faster than doubles, but after doing some research I've found that there's some confusion about…
Trap
  • 12,050
  • 15
  • 55
  • 67
54
votes
5 answers

How to calculate precision and recall in Keras

I am building a multi-class classifier with Keras 2.02 (with Tensorflow backend)，and I do not know how to calculate precision and recall in Keras. Please help me.
Jimmy Du
  • 545
  • 1
  • 4
  • 6
48
votes
3 answers

Changing precision of numeric column in Oracle

Currently I have a column that is declared as a NUMBER. I want to change the precision of the column to NUMBER(14,2). SO, I ran the command alter table EVAPP_FEES modify AMOUNT NUMBER(14,2)' for which, I got an error : column to be modified…
roymustang86
  • 8,054
  • 22
  • 70
  • 101
48
votes
7 answers

How to create a high resolution timer in Linux to measure program performance?

I'm trying to compare GPU to CPU performance. For the NVIDIA GPU I've been using the cudaEvent_t types to get a very precise timing. For the CPU I've been using the following code: // Timers clock_t start, stop; float elapsedTime = 0; // Capture…
sj755
  • 3,944
  • 14
  • 59
  • 79
1 2
3
…
99 100