Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.
Questions tagged [floating-point-precision]
551 questions
0
votes
1 answer
Integer multiplied by float and then divided by same float - a clean way to force value back to original integer
Alright so we have the classic 1 * 1/3 * 3 = 0.9999999. I understand why this is the case.
In my use case, I am trying to result in an integer. I am happy to take the floor of my value, except when infinite precision math dictates that we should…
user321605
- 846
- 2
- 7
- 20
0
votes
1 answer
adding decimal numbers with JavaScript
I hate to add to the litany of questions regarding this subject but I dont know where else to turn for a concise answer so here goes:
I have this form on jsFiddle
I got to this point by combining several different examples I have found online today…
Wesley Smith
- 19,401
- 22
- 85
- 133
0
votes
1 answer
bcadding very small floats gives 0
Thanks to stack, I learned about floating point imprecision, so I went over to the bc functions.
That works great on "normal" floats, but with extremely small floats, say 10^-10 types, bcadd always gives 0.
Can someone show me what I'm doing wrong…
user1382306
0
votes
2 answers
MIL-STD-1750A to Decimal Conversion Examples
I am looking at some examples in the 1750A format webpage and some of the examples do not really make sense. I have included the 1750A format specification at the bottom of this post in case anyone isn't familiar with it.
Take this example from…
halexh
- 3,021
- 3
- 19
- 19
0
votes
2 answers
adjusting floats to maintain high precision
I am writing some Python code that requires a very high degree of precision. I started to use Numpy float64, but that didn't work as required, and I then started using the "Decimal" module, which then worked fine.
I would ideally prefer, however, to…
user1551817
- 6,693
- 22
- 72
- 109
0
votes
2 answers
floating points precision in complicated calculations
For calculating Catalan Numbers, I wrote two codes. One (def "Catalan") works recursively and returns the right Catalan Numbers.
dicatalan = {}
def catalan(n):
if n == 0:
return 1
else:
res = 0
if n not in dicatalan:
for i in…
jizanthapus
- 121
- 1
- 9
0
votes
1 answer
Matlab import CSV file with header and float precision numbers
My problem is to import numbers inside CSV file to matlab variable with the same precision as original number. Here's an example of my data (there can be X amount of rows with one header on top of file):
g_B_esti_Velx, g_B_Velx, g_B_ACCx,…
ajr
- 874
- 2
- 13
- 29
0
votes
2 answers
How to make numbers not be shown in scientific form?
I want to write an array of floating point numbers into files
but in the test.txt, I…
user1769686
- 505
- 2
- 10
- 16
0
votes
1 answer
wrong calculation result in Xcode
I have this calculation:
float tempd = 2451545 + 0.0009 + (77.048759/360) + 4682;
NSLog(@"tempd : %f", tempd);
The result using Xcode is : 2456227.250000
But it should be : 2456227.21492
Do you guys have any idea what I'm doing wrong?
I cant seem…
Falcko
- 115
- 1
- 1
- 10
0
votes
0 answers
libquadmath and GNU-gsl ode solver?
I would like a working precision of 40 or more decimal places whilst solving an ODE.
Could anyone tell me if it is possible to get higher precision results using the gsl ODE solver, perhaps using libquadmath? It looks from the gsl_odeiv2…
fpghost
- 2,834
- 4
- 32
- 61
0
votes
0 answers
XNA large coordinates and float precision
I have my model created in Autocad as .dwg file in large coordinates like (X = 528692.833, Y = -261.184, Z = 1.890). When the model is exported to .Fbx file, the distortions appear in the FBX Viewer and the same happens in XNA with converted .Xnb…
ali
- 529
- 4
- 26
0
votes
1 answer
Removing the trailing zeros in a double variable
i have a DLL which defines many methods which all returns double and I don't want to change anything in it. The problem is when I pass small numbers to these methods, the return values come with too many trailing zeros as the return value is double,…
Eslam Hamdy
- 7,126
- 27
- 105
- 165
0
votes
1 answer
Decimal-based Complex struct; A good idea?
I have a hankering to go home tonight and bodge up a Mandelbrot/Julia fractal generator. It's a project I've undertaken before in C++, but this time I'll give it a whirl in C# so I can more easily make it a WinForms app, use multithreading to…
KeithS
- 70,210
- 21
- 112
- 164
0
votes
2 answers
How to convert floating point input to integers and preserve maximum precision?
I have to use an algorithm which expects a matrix of integers as input. The input I have is real valued, therefore I want to convert the input it to integer before passing it to the algorithm.
I though of scaling the input by a large constant and…
stressed_geek
- 2,118
- 8
- 33
- 45
0
votes
1 answer
How to "append" significant figures to double from float value
Currently I am trying to match a C# and C++ application. On the C++ side, when there is a value, say:
const char* svalue = "554.1327";
When I use sscanf:
float x;
sscanf(svalue, "%f", &x);
x will equate to 554.13269, i.e. it has "additional"…
