Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
0
votes
1 answer

Achieving intended result from fmod or similar in spite of floating point arithmetic

Background Consider the validation of three doubles low, width and high such that the following three rules hold true: low < high; width > 0; and width fits into (high - low) "exactly" a whole number of times. Essentially, the three values should…
JimmidyJoo
  • 10,503
  • 7
  • 27
  • 30
0
votes
1 answer

Real numbers comparison

Possible Duplicate: Comparing floating point values How dangerous is it to compare floating point values? I don't understand, why comparison of real numbers is a bad practice in programming? Of course I understand that real numbers can be…
Sergey Shafiev
  • 4,205
  • 4
  • 26
  • 37
0
votes
3 answers

decimal type - odd behaviour displaying to the console with redundant zeros when parsing from a string

We are finding some strange behaviour with the decimals being displayed to the console. This is best discribed by the following code: string num1AsString = "1.0000"; decimal num1AsDecimal = decimal.Parse(num1AsString); string num2AsString =…
bstack
  • 2,466
  • 3
  • 25
  • 38
0
votes
0 answers

PHP's Float Precision Causing Problems With Small Numbers

I have an ojbect $order that has a member function amount() that adds up the sum of the line items (all floats) then returns the value (again, as a float). var_dump(array( $order->amount() * 100.0, intval($order->amount() *…
Ian
  • 24,116
  • 22
  • 58
  • 96
0
votes
0 answers

Scrubbing code to enable -ffinite-math

I have a couple audio libraries that I need to scrub so I can turn on -ffinite-math. I know I need to remove the use of NaN and +/-Inf as sentinel values but am unclear how to ensure stability in the code. I am also not sure what kind of floating…
user1332148
  • 1,256
  • 2
  • 11
  • 24
0
votes
2 answers

Double precision - MS visual C++ 2005 Vs Matlab

I am using MS visual C++ 2005 for implementing some mathematical algorithms in C++. My C++ code takes 10K double data type input values with precision of 12 decimal places(For e.g. 866.333333333333) which are generated in Matlab. Then my code does…
Yogi
  • 83
  • 1
  • 10
-1
votes
2 answers

Java Float multiplication giving wrong result

Multiplication using FLOAT is giving noticeable difference. public static void main(String[] args) { // using string and parsing instead of actual data type is part of use case, that is why representing the same here double v1 =…
jagadesh
  • 71
  • 1
  • 6
-1
votes
1 answer

Relation between Cnn Modell and Single precision fp32/int8

I'm currently writing my BA Thesis about Hardware and Frameworks for AI-Inference. In my research, I looked up TensorRT and find ab table which I don't really understand. Table Sadly there is no real explanation for this except the title. I…
Morty687
  • 1
-1
votes
1 answer

Reduce or solve calculative errors in Javascript

Sometimes simple algebra can result in floating accuracy errors. in this case, I encountered 12 * 1.6 which resulted in 19.20000000003 Is there a way to catch this issue or prevent it from happening? Accuracy is highly important so simply rounding…
DIGI Byte
  • 4,225
  • 1
  • 12
  • 20
-1
votes
1 answer

pandas mess up multi level index parquet float accuracy

I have some data that has multi level index & column. My goal is to store the data as float32 instead of double/float64 into parquet file, to save disk space. I don't care about the accuracy of the cell values, but I do want the index & column…
abisko
  • 663
  • 8
  • 21
-1
votes
1 answer

Python Decimal Library is Imprecise?

I'm reading up on the Python Decimal module. I have a need to make a large number of precise calculations, often with lots of decimal places, where being off by a small amount adds up over time. Enter the Decimal library. Step 1: Read the intro to…
scotty-nukem
  • 21
  • 3
-1
votes
1 answer

Convert a continuous range of doubles to a discrete int range

i want to map a continuous range of double values [10.0,20.0] to a byte in the range of [100,200]; The following must apply (even when converting back and forth multiple times): convertToByte(convertToDouble(y)) == y Thanks.
zel873ju
  • 139
  • 2
  • 7
-1
votes
1 answer

Concept related to precision of float and double

Why the precision of the float is up to 6 digits after the decimal point and the precision of the double is up to 15 digits after the decimal point? Can anyone give a mathematical explanation of it?
Hash Blog
  • 19
  • 2
-1
votes
2 answers

Why does python get confused by these comparisons?

If I run this code in Python 3.8.6 I get the output False: (((3980 - 91 + 1)/(3980)) * ((1 / 3980)**(91 - 1))) > (((3981 - 91 + 1)/(3981)) * ((1 / 3981)**(91 - 1))) However, if I run this in WolframAlpha it returns True. I'm guessing this happens…
Connor
  • 4,216
  • 2
  • 29
  • 40
-1
votes
1 answer

java-processing floating point rounding error, how to keep radians rational

I have modified this arcball class so that every call to arcball.rollforward(PI/180); rotates a matrix 1 degree. I have tried to set it up so arcball.rollback() is called with the accumulated float rotatebywithincludedfloaterror but it has had the…
cubesareneat
  • 302
  • 2
  • 14
1 2 3
…
90 91