Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
13
votes
2 answers

getting the average, p95 and p99 of a stream of data

I have incoming data and I want to compute the average, 95th and 99th percentile of that data - I am most interested in the last 1000 values. At any time, I'd like to query this object to get any of the three values (this can occur at any time, not…
jamesatha
  • 7,280
  • 14
  • 37
  • 54
13
votes
2 answers

C++ calculating more precise than double or long double

I'm teaching myself C++ and on this practice question it asks to write code that can calculate PI to >30 digits. I learned that double / long double are both 16 digits precise on my computer. I think the lesson of this question is to be able to…
Terence Chow
  • 10,755
  • 24
  • 78
  • 141
13
votes
2 answers

print double with precision 4 using cout

Possible Duplicate: Convert a double to fixed decimal point in C++ Suppose , I have double a = 0 and I want to print it as 0.0000 . I've tried this : cout.precision(4) ; cout<
URL87
  • 10,667
  • 35
  • 107
  • 174
13
votes
3 answers

SQL set floating point precision

For a SQL int that is being converted to a float, how do I set the precision of the floating point number? This is the selection I would like to truncate to two or 3 decimal places: AVG(Cast(e.employee_level as Float))avg_level, Thanks!
CodeKingPlusPlus
  • 15,383
  • 51
  • 135
  • 216
13
votes
1 answer

Subtraction without overflow?

Suppose that there are two integers(int x, y;). x is negative and y = 0x80000000. Why does (x - y) not overflow while x + (-y) does? Doesn't the computer do subtraction by addition?
Yuu
  • 147
  • 6
13
votes
2 answers

Why is -freciprocal-math unsafe in GCC?

-freciprocal-math in GCC changes the following code double a = b / c; to double tmp = 1/c; double a = b * tmp; In GCC manual, it's said that such an optimization is unsafe and is not sticked to IEEE standards. But I cannot think of an example.…
Kid
  • 305
  • 1
  • 5
12
votes
5 answers

Converting 8 bytes of little-endian binary into a double precision float

I have a binary file that I read byte by byte. I come across a section that is 8 bytes long, holding a double precision float (little endian). I can't figure out how to read this in and calculate it properly with masking and/or casting. (To be…
RedLeader
  • 657
  • 1
  • 15
  • 28
12
votes
2 answers

Instability in DeleteDuplicates and Tally

While preparing an answer to Count how many different values a list takes in Mathematica I came across an instability (for lack of a better term) in both DeleteDuplicates and Tally that I do not understand. Consider first: a = {2.2000000000000005,…
Mr.Wizard
  • 24,179
  • 5
  • 44
  • 125
12
votes
5 answers

Efficient stable sum of ordered numbers

I have a quite long list of floating point positive numbers (std::vector, size ~1000). The numbers are sorted in decreasing ordering. If I sum them following the order: for (auto v : vec) { sum += v; } I guess I can have some numerical…
Ruggero Turra
  • 16,929
  • 16
  • 85
  • 141
12
votes
4 answers

Why is "(2.5 < 2.5 + Number.EPSILON)" false in JavaScript?

I want to find values less than a certain value in the array. I tried to use Number.EPSILON because the input value is not a definite value (eg 1.5000000000001). I found something strange during the test: >> (1.5 < 1.5 + Number.EPSILON) <- true…
Wakeup
  • 155
  • 1
  • 7
12
votes
4 answers

Reducing the precision of numbers - regex vim

My regex is quite rusty. How could vim be used to change the precision of a decimal number. For example: Changing 30.2223221 8188.2121213212 to 30.22 8188.21
Aiman
  • 123
  • 1
  • 4
12
votes
4 answers

What is the role of **std::setprecision()** without **std::fixed** in c++?

As shown in the tutorial http://www.cplusplus.com/reference/iomanip/setprecision/ // setprecision example #include // std::cout, std::fixed #include // std::setprecision int main () { double f =3.14159; std::cout…
Talespin_Kit
  • 20,830
  • 29
  • 89
  • 135
12
votes
1 answer

Working with small probabilities, via logs

Source: Google Code Jam. https://code.google.com/codejam/contest/10224486/dashboard#s=a&a=1 We're asked to calculate Prob(K successes from N trials) where each of the N trials has a known success probability of p_n. Some Analysis and thoughts on the…
Brondahl
  • 7,402
  • 5
  • 45
  • 74
12
votes
3 answers

Spreadsheet calculations with (at least) the accuracy of a C double

I was doing some calculations for planning an improved implementation of my prime sieve when I noticed that the Libre Office spreadsheet was showing the wrong values for numbers far below 2^53, which is the limit for precise integer calculations in…
DarthGizka
  • 4,347
  • 1
  • 24
  • 36
12
votes
4 answers

Firing events at microsecond resolution for midi sequencer

Is there a way to fire events in C# at a resolution of a few microseconds? I am building a MIDI sequencer, and it requires an event to be fired every MIDI tick, which will then play any note registered at that time. At 120 beats per minute and at a…
Brice
  • 535
  • 6
  • 11
1 2 3
…
99 100