Stack Overflow
Stack Overflow

Questions tagged [floating-accuracy]

Concerning the accuracy of operations performed on floating point numbers.

Floating point numbers (typically meaning the IEEE standard) are inherently inexact and errors can compound, leading to edge cases in some decision processes, or numerical instability in certain algorithms.

Here is a mathematical treatment of the main problems.

1358 questions
0
votes
1 answer

IEEE floating points implementation, precision and accumulation of approximations

If I understand IEEE floating points correctly, they are unable to accurately represent some values. They are accurate in very limited cases and pretty much every floating point operation increases the accumulated approximations. Also, another…
user2341104
0
votes
2 answers

Round float values in JavaScript

I am trying to round a value in JS but I am getting not rounded value, this is what I have: $(document).on("click", ".open-AddBookDialog", function () { var agentPercentage =…
Laziale
  • 7,965
  • 46
  • 146
  • 262
0
votes
1 answer

A more accurate data type than float or double? C++

Does C++ have a more accurate data type than float or double, or do I just have to settle for the fact that my calculations will be off? EDIT: As Mr. Lister has pointed out, my question is regarding the precision. It's a bit frustrating when you…
B.K.
  • 9,982
  • 10
  • 73
  • 105
0
votes
4 answers

Fix for float precision issue without BigDecimal

I'm having an issue with the rounding of a float value. The following code gives me the following result: public class ProductOrder { public static void main(String[] args) { int q = 48; float p = 6.95f; System.out.println(q * p); …
wvp
  • 1,134
  • 5
  • 15
  • 28
0
votes
1 answer

floating point values read from file and calculations discrepancy

I have dll which reads floating point values from binary files, makes simple calculation and gives boolean result. Each file is 8bytes length so variables are of type double ( Visual Studio 2008 ). Calculation is very simple: if( fA < fB - ( iX *…
psu
  • 111
  • 1
  • 10
0
votes
3 answers

'Beautify' number by rounding erroneous digits appropriately

I want my cake and to eat it. I want to beautify (round) numbers to the largest extent possible without compromising accuracy for other calculations. I'm using doubles in C# (with some string conversion manipulation too). Here's the issue. I…
Dan W
  • 3,520
  • 7
  • 42
  • 69
0
votes
1 answer

ruby script using sqlite database decimal set to (4,2) but contains value (10,8)

I have a sql query that created a table like so: db.execute ("CREATE TABLE IF NOT EXISTS RATING (programCode INTEGER , fromDate INTEGER, toDate INTEGER, programName TEXT, channelName TEXT, weekday TEXT, startTime TEXT, endTime TEXT, duration…
banditKing
  • 9,405
  • 28
  • 100
  • 157
0
votes
1 answer

How can SQLServer represent 0.2 as FLOAT

DECLARE @f as float SELECT @f = 0.2 SELECT @f The above code returns ---------------------- 0.2 (1 row(s) affected) No surprise there. Except that floating-point cannot represent 0.2 exactly. So what is SQLServer doing here? Is it using…
El Ronnoco
  • 11,753
  • 5
  • 38
  • 65
0
votes
2 answers

How to weigh up calculation error

Consider the following example. There is an image where user can select rectangular area (part of it). The image is displayed with some scale. Then we change the scale and we need to recalculate the new coordinates of selection. Let's take…
x-yuri
  • 16,722
  • 15
  • 114
  • 161
0
votes
1 answer

MATLAB: Save command not working

I am trying to save some variables at different time steps in a while loop using the 'save' command. It was working in other programs which I had written previously. This is the first time it's not saving. I tried many things like renaming the file…
koolraj09
  • 13
  • 2
  • 7
0
votes
0 answers

Python: floating accuracy when computing RTTs

With tcpdump, I dumped a trace of around 2800 outgoing packets and just as many corresponding incoming ICMP packets. Then, I computed the RTT for each outgoing packet with a simple subtraction. It turns out that, even though there are no duplicates…
Ricky Robinson
  • 21,798
  • 42
  • 129
  • 185
0
votes
2 answers

how to avoid inaccuracy in multiplications and divisions in python

I am working with python and playing with multiplications and divisions but I can't get the accuracy that I want with this operations. The following example may help to reach my point. Let a = 3 and b = 4, their euclidean norm (sqrt(a*a + b*b)) is…
Alejandro Sazo
  • 796
  • 15
  • 31
0
votes
2 answers

Accuracy of c_k = a + ( N + k ) * b

a, b are 32 bit floating point values, N is a 32 bit integer and k can take on values 0, 1, 2, ... M. Need to calculate c_k = a + ( N + k ) * b; The operations need to be 32 bit operations (not double precision). The concern is accuracy -- which…
user1823664
  • 1,071
  • 9
  • 16
0
votes
2 answers

Rounding off floating point values

While rounding off the floating point values I observed some discrepancy in values. I have extracted following part of code. Here if variable var_d is assigned value> 5.3 to then I am getting proper values for variable var_d, but for values like…
aashoo
  • 404
  • 4
  • 13
0
votes
2 answers

Why is "long double" type of a variable producing absurd output while "float" and "double" type works fine?

All I want the following program to do is to print the product of all positive numbers less than 30 (in both exponential and non-exponential forms). It works fine when the variable product is declared as a float or a double,but produces totally…
Rüppell's Vulture
  • 3,583
  • 7
  • 35
  • 49
1 2 3
…
90 91