Stack Overflow
Stack Overflow

Questions tagged [precision]

For questions related to numerical precision in programming. For classification precision use the tag [precision-recall].

In general, the precision of an approximate real number is the effective number of decimal digits in it which are treated as significant for computations. The accuracy is the effective number of these digits which appear to the right of the decimal point.

For precision in Information Retrieval (the ratio of relevant items to total items retrieved) please use the tag .

4478 questions
2
votes
0 answers

How to detect if a result is due to an approximation in floating division?

From a data set I'm working on I have produced the following graph: The graph it is constructed as follows: to each element on the data set it is associated the ratio between two natural numbers (some times big numbers) where the numerator is…
user2988577
  • 3,997
  • 7
  • 21
  • 21
2
votes
2 answers

How to increase precision of double type for large numbers?

In this sample code double number = 2646849856355434345; Console.WriteLine( number % 5 ); If I use Double for this operation, it won't give a precise answer. Here it will output "1" where it should output "0". Is there anyway to increase the…
Ayman El Temsahi
  • 2,600
  • 2
  • 17
  • 27
2
votes
1 answer

Is there any standard library to convert float64 to string with fix width with maximum number of significant digits?

Imagine for printing in a 12 fixed width table we need printing float64 numbers: fmt.Printf("%12.6g\n", 9.405090880450127e+119) //"9.40509e+119" fmt.Printf("%12.6g\n", 0.1234567890123) //" 0.123457" fmt.Printf("%12.6g\n", 123456789012.0) …
user6169399
2
votes
3 answers

Taking Modulo of Double NUmber

I have given two number a and b.I have to Calculate (a^b)%1000000007.How Can i calculate for floating point numbers. Ex: a= 7.654 and b=10000 Here is my Code will % work : public static double super_pow(double A , long B){ double o=1; …
Narendra Modi
  • 851
  • 1
  • 13
  • 29
2
votes
0 answers

Python, C++, and quad precision

I'm currently trying to take some C++ code and interface it with Python. I'm aware there are a few different ways to do this. Cython, Boost Python, and the package I'm most interested in is pybind11. I'm most interested in pybind11 because it…
mike van der naald
  • 259
  • 2
  • 11
2
votes
1 answer

Why fix((20.45-20)*60) is not equal with fix((20)*60) in MATLAB

There is an interesting question on Facebook which I encountered. He asks that he is using MATLAB R2014a (64-bit version) and ran this code:- x = 0.45; min = fix((x)*60); fprintf('min = %d \n', min); x = 20.45 min = fix((x-20)*60); fprintf('min =…
Thammarith
  • 662
  • 5
  • 19
2
votes
1 answer

When to use dsgesv versus dgesv to solve a system of linear equations

In the LAPACK documentation, it states that DSGESV (or ZCGESV for complex numbers) is: The dsgesv and zcgesv are mixed precision iterative refinement subroutines for exploiting fast single precision hardware. They first attempt to factorize the…
EMiller
  • 2,792
  • 4
  • 34
  • 55
2
votes
3 answers

unexpected difference between types results

#include #include using namespace std; string number = "159"; float valueFloat = 0; int valueInt = 0; int main() { for(int i = number.length()-1; i >=0; i--) { valueInt += (number[i]-48) * pow(10,i); …
DawidChrzest
  • 46
  • 4
2
votes
2 answers

Java - Practical difference between 2.0 and 2.00000000000D

I'm rewriting some scientific code that someone else wrote a while back, and throughout the code constants are always declared as: final double value = 2.0000000000D; with an apparently arbitrary length of supposedly significant digits attached to…
user1337732
  • 68
  • 5
2
votes
5 answers

haskell floating point precision

I wrote a little program to find the position of a point(x,y) in relation to a line defined by a point(px,py) and an angle(deg) to the x-axis (in a cartesian coordinate system). toRadian deg = deg * (pi / 180) lineSlope deg = tan $ toRadian…
anybody
  • 121
  • 8
2
votes
2 answers

Division of double-precision by an integer in Fortran

Right now I am dealing with Fortran codes of the man who worked on the project before me. The code is relatively huge so I would not provide it here. In a lot of places in this codes there are division of double precision value over integer.…
Gogrik
  • 48
  • 1
  • 1
  • 6
2
votes
2 answers

Accuracy of Adding Floats vs. Multiplying Float by Integer

In my computer science course, we are doing a study of floating point numbers and how they are represented in memory. I already understand how they are represented in memory (the mantissa/significand, the exponent and its bias, and the sign bit),…
Spencer D
  • 3,376
  • 2
  • 27
  • 43
2
votes
2 answers

Float with single dot using Regex allowed and replacing the value with the valid value if it is invalid value

I want that a text field should allow float numbers with single dot and If I am copying from somewhere it will replace it and make it a float number. but below code is working only for number $("#max_width_value").bind("change keyup input",…
dixit thareja
  • 83
  • 1
  • 11
2
votes
0 answers

Precision and Scale max limits for Float and Double precision types in SqlServer

What are the limits of the precision and scale for Float and Double datatypes.Float allows scale of max value of, what are the restrictions for it.suppose if we give Float(19) but still it can store values having scale like (2555555.55555555). and…
anusha dharmasagar
  • 21
  • 4
2
votes
1 answer

numpy `arange` exceeds end value?

I had expected numpy's arange(start,end) to produce values in the range [start,end]. The following example demonstrates that that's not always true (the final value is larger than end): import numpy as np start=2e9 end=start+321 …
Bill Bradley
  • 456
  • 4
  • 16
1 2 3
…
99
100