Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
2
votes
2 answers

Addition vs Subtraction in loss of significance with floating-points

While learning about precision in floating point arithmetic and different methods to avoid it (using a conjugate, taylor series,...) books frequently mention the subtraction of two very similar numbers or one large and one small number as the…
ZuluForce
  • 217
  • 2
  • 6
2
votes
4 answers

Big numbers in c++ issue in cosine similarity

I'm writing this function double long CosineDistance(const vector& a,const vector& b){ double long num = 0.0, den1 = 0.0, den2 = 0.0 ; for(int i = 0; i < a.size(); ++i) { num+=a[i]*b[i] ; den1+=a[i]*a[i]…
Ghesio
  • 89
  • 11
2
votes
1 answer

how single and double type variables work in the same copy of code in Matlab like template in C++

I am writing a signal processing program using matlab. I know there are two types of float-pointing variables, single and double. Considering the memory usage, I want my code to work with only single type variable when the system's memory is not…
Hua
  • 184
  • 8
2
votes
3 answers

Why does nextUp method in Math class skips some values?

I was just messing around with this method to see what it does. I created a variable with value 3.14 just because it came to my mind at that instance. double n = 3.14; System.out.println(Math.nextUp(n)); The preceding displayed…
Haggra
  • 3,539
  • 2
  • 20
  • 28
2
votes
3 answers

Convert Float value to two point decimal

I need to find the way for converting the Float value in to the two point decimal. I have a API code which is implemented already which will be returning the two Float value. For example consider it is returning the values as follows:…
Sparky
  • 91
  • 1
  • 8
2
votes
1 answer

Use of half2 in CUDA

I am trying to use half2, but I run into an error, namely, error: class "__half2" has no member "y" The section of code where the error occurs is as follows: uint8_t V_ [128]; // some elements (uint8), to save space float V_C[128]; //…
manyids2
  • 23
  • 1
  • 6
2
votes
1 answer

precision num2string matlab

I got some question relating the precision of the num2str() fctn in MatLab. a=0.11111111111111; b=a; Linux/OSX: num2str(a+b,25): ans=0.2222222222222221655465116 Windows: num2str(a+b,25): ans= 0.222222222222222 Can anyone explain where the numbers…
moatze
  • 148
  • 2
  • 3
  • 12
2
votes
2 answers

Why doesn't this produce a double underflow?

Can anyone explain why this snippet does not produce an underflow exception (on MSVC 2013 and on gcc @ coliru)? The value returned from the average function is lower than DBL_MIN. #include #include #include #include…
Rudolfs Bundulis
  • 11,636
  • 6
  • 33
  • 71
2
votes
1 answer

Why does eps fail in a matrix when used with realmax

See >> eps([1 2 0.00001; (realmax('double')-10) realmin('double') realmax('single')]) ans = 1.192093e-07 2.384186e-07 9.094947e-13 NaN 1.401298e-45 2.028241e+31 However, >> eps(realmax('double') - 10) ans = …
Alex
  • 15,186
  • 15
  • 73
  • 127
2
votes
1 answer

Floating Point Number issue JavaScript

I'm struggling to resolve a Floating Point Number issue where var change returns as 0.0999 recurring, and i need to return 0.01 (one penny). The code works fine, except the very last penny, because of this issue. This is my first post here so please…
Andy Wallace
  • 23
  • 4
2
votes
1 answer

OpenGL: Is the precision of GL_R11F_G11F_B10F good for 8bit RGB images?

From "Image Format", we see: An 11-bit float has no sign-bit; it has 6 bits of mantissa and 5 bits of exponent. From "Half-precision floating-point format", we can deduct that: The relative precision of 11-bit float is 2^(-6), which is 1/64 But…
user1914692
  • 3,033
  • 5
  • 36
  • 61
2
votes
0 answers

16-bit floating point on fpga

I try to use Altera's floating point IP to generate half precision instead of single (32-bit) blocks for addition , multiplication etc. However when configuring the IP it seems that half precision fp needs a lot more LUTs and has far more latency in…
user2609910
  • 157
  • 8
2
votes
1 answer

Mysqli prepared statement returned float value

I've been trying to solve this problem for the whole day, and no solution yet. I have a test bench on windows using XAMPP, when I retrieve a float value from the database, I get the value like this: mysqli normal query : 0.67 mysqli prepared stmt:…
Osama Mohammed
  • 129
  • 6
2
votes
1 answer

Multiplication of floating point numbers gives different results in Numpy and R

I am doing data analysis in Python (Numpy) and R. My data is a vector 795067 X 3 and computing the mean, median, standard deviation, and IQR on this data yields different results depending on whether I use Numpy or R. I crosschecked the values and…
London Holmes
  • 79
  • 10
2
votes
0 answers

Using J# Redistributable in C#

I'm trying to get an arbitrary-precision decimal class in C# as per this answer. I've downloaded and installed the J# re distributable, but I'm clueless as to how to import / reference it into my visual studio project. The folder it has created…
acernine
  • 729
  • 2
  • 6
  • 16
1 2 3
…
99
100