Stack Overflow
Stack Overflow

Questions tagged [floating-point-precision]

Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.

551 questions
2
votes
3 answers

How to count decimal places of float?

I want to check if a float32 has two decimal places or not. My javascript way to do this would look like: step := 0.01 value := 9.99 if int(value/step) % 1 == 0 { printf("has two decimal places!") } The above example also works. However it…
bodokaiser
  • 15,122
  • 22
  • 97
  • 140
2
votes
3 answers

Perl: Which floating point operations are lossless?

There are some great docs out there describing the nature of floating point numbers and why precision must necessarily be lost in some floating point operations. E.g.,: http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html I'm interested in…
Vijay Boyapati
  • 7,632
  • 7
  • 31
  • 48
2
votes
2 answers

Lower the floating-point precision in python to increase performance

I'm working with python on raspberry pi. I'm using complementary filter to get better values from gyroscope, but it eats too much raspberry's power - it's about 70%. I thought I could increase performance by reducing floating point precision. Now,…
ametis
  • 41
  • 5
2
votes
0 answers

C++ precision and truncation with file stream

I have a file.txt with hundreds of numbers. They have many digits (max 20) after the point and I need to get them all without truncation, otherwise they introduce errors in the following computations. I made these numbers with matlab so it has a…
user3290180
  • 4,260
  • 9
  • 42
  • 77
2
votes
1 answer

Decrement a float from 1 down to 0, in 0.01 increments

As the title states, I'd like to start from 1 and decrement with by 0.01 all the way down to zero. Problem is, I'm using floats and I keep getting values such as 0.5000000001.
Ogen
  • 6,499
  • 7
  • 58
  • 124
2
votes
4 answers

Python: only two decimal places after user entered number

How do you make sure that the user only entered a float that has two decimal places. The number cannot be 4 or 4.999 it has to be 4.00 or 4.99 otherwise an error should appear. while looping: try: number = float(input("Number: ")) …
novice coder
  • 33
  • 2
  • 6
2
votes
3 answers

Error with Random.nextDouble math

I am working on a small app to deposit and withdraw funds from a bank account. The program is running mostly fine, but I am getting strange output from some of the transactions with a lot of trailing decimals. I am randomly generating doubles that…
Jonathon Anderson
  • 1,162
  • 1
  • 8
  • 24
2
votes
2 answers

Error while trying to use polyval

I have the following vector vec = [ 255 0 255 0 255 0 255 0 255 0 255 0 255 0 255 0] vec 1x16 double and using the following command polyval(vec', 256); I get ans = 3.3896e+038 but when I try to get…
ealiaj
  • 1,525
  • 1
  • 15
  • 25
2
votes
2 answers

Anyone know why matlab would show many significant digits for no reason?

Example command and output: >> 15.12 ans = 15.119999999999999" I'm trying to show only two significant digits (I'm putting values into a uitable) so this is driving me crazy.
user1949902
  • 361
  • 3
  • 9
2
votes
2 answers

using real(A[,kind=]) in passing a single precision array as a double precision to a subroutine in Fortran 90/2003

I need to perform both single precision and double precision arithmetics on a variable in different parts of my code. So basically, I declare the variable as single precision first. Then I call subroutine sub_a which makes use of double precision…
Amir
  • 335
  • 2
  • 11
2
votes
1 answer

Floor functions in cmath precision error

So I was recently trying a problem involving floating point arithmetic. Below is a snippet of that code. #include #include typedef unsigned long long ull; int main(){ long double n=1000000000,ans,ans1,ans2; ans =…
Jignesh
  • 555
  • 1
  • 8
  • 18
2
votes
1 answer

Floating point addition and division

For this code segment double count = 0.0; while( count != 1.0) {count += 1.0/3;} I was wondering what parts of (IEEE 754) would cause count 3: to be 1.0 instead of .9999999999999999. I realize that the general reason would be rounding due to an…
user3361646
  • 31
  • 1
2
votes
2 answers

Why does FloatToText return different values in different projects?

I'm using FloatToText this way: function ExFloatToStr(Value: Extended): string; var Buffer: array[0..63] of Char; FormatSettings: TFormatSettings; begin GetLocaleFormatSettings(GetUserDefaultLCID, FormatSettings); SetString(Result, Buffer,…
taoxl
  • 125
  • 1
  • 1
  • 5
2
votes
1 answer

Floating-point error when checking for coplanar 3D points

I am looking for an algorithm to check if a point is coplanar with a given 3D plane, defined out of three vertices, while minimizing floating point errors. I would like to minimize the amount of multiplications and division to mitigate floating…
Julien-L
  • 5,267
  • 3
  • 34
  • 51
2
votes
2 answers

Using floats for counters - Any problems?

The code I'm looking at uses a golang float64 as a counter, does this pose a problem with loss of accuracy at some point? Specifically, are all integers represented in the range covered by float64? If not, where do I start running into problems?…
JnBrymn
  • 24,245
  • 28
  • 105
  • 147
1 2 3
…
36 37