Stack Overflow
Stack Overflow

Questions tagged [floating-point-precision]

Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.

551 questions
2
votes
3 answers

Typecasting int to float turns DivideByZeroException into Infinity

I wrote a simple divide function in C#: private string divide(int a, int b) { return string.Format("Result: {0}", a / b); } Calling MessageBox.Show(divide(3, 0)) results in, as you would expect, a DivideByZeroException. So I…
ashes999
  • 9,925
  • 16
  • 73
  • 124
2
votes
3 answers

Large number of float digits without extra library

i have a float value that is hundreds of digits long (like the first 100 digits of pi - 3) and need a way to operate on it. is there any way to store and operate on the float that has a large number of decimals and maintain much precision with built…
cspotrun
  • 21
  • 1
  • 2
2
votes
1 answer

How to factorize a big integer

I am trying to factorize a big integer with the following code: library(gmp) as.bigz(factorize( 113423713055421844361000443)) Big Integer ('bigz') object of length 38: # [1] 2 2 2 2 2 2 2 2 2 2 …
aarthi
  • 85
  • 5
2
votes
1 answer

How to avoid rounding off of large float or double values?

I am working on some c++ problem which should output 22258199.5000000 this number, I have stored the result in double datatype and when I tried to print that variable using std::cout it rounds off to 22258200.0000000, I don't know why is this…
ankit deora
  • 79
  • 1
  • 7
2
votes
0 answers

Android ENCODING_PCM_FLOAT Audiotrack issue

I'm writing an app to create a sound with depth to 32-bit audio, but can not understand much. I have a code that generates the tone and another that generates the audiotrack. I'm new in this topic, and I have not found what I ask from anywhere, so I…
Northumber
  • 315
  • 2
  • 3
  • 15
2
votes
1 answer

How to obtain only the integer part of a long floating precision number with JS?

I know there's Math.floor parseInt But about this case: Math.floor(1.99999999999999999999999999) returning 2, how could I obtain only its integer part, equals to 1?
diegoaguilar
  • 8,179
  • 14
  • 80
  • 129
2
votes
2 answers

awk read float precision: cannot read small floats, e.g. 4e-320

I cannot get awk or gawk to read small floats in scientific notation and interpret them correctly as floating point numbers. I just want to output numbers above a small threshold with awk. Example: consider the following…
cmo
  • 3,762
  • 4
  • 36
  • 64
2
votes
1 answer

Numpy sum function returns 1.67772e+07

I have two big (432*136*136*46) 'numpy.ndarray' H1 and H2 which encompass altitude values corresponding to two simulations. I want to generate an array with 1 when H1 and H2 have the same altitude and 0 when they don't. Then, I want to know how many…
Arnaud BUBBLE
  • 677
  • 1
  • 6
  • 10
2
votes
1 answer

Why are these two calculations which are exactly same giving different results in Fortran using gfortran?

real, dimension(3), parameter :: boxlen = [4.0, 5.0, 7.0] real, parameter :: mindist = 0.1 integer ::i write(*,"(A)") "Operation on array" print*, floor(boxlen/mindist) write(*,"(/A)") "Operation on individual elements" do i=1,3 …
Yogesh Yadav
  • 404
  • 5
  • 16
2
votes
2 answers

Generating evenly distributed bits, using approximation

I'm trying to generate 0 or 1 with 50/50 chance of any using random.uniform instead of random.getrandbits. Here's what I have 0 if random.uniform(0, 1e-323) == 0.0 else 1 But if I run this long enough, the average is ~70% to generate 1. As seem…
f.rodrigues
  • 3,499
  • 6
  • 26
  • 62
2
votes
1 answer

php - bcadd has different point precision

On my windows box when I run $SR = "0"; $SPR = "149"; $SR = bcadd($SR, $SPR); echo "$SR"; It outputs 149.0000000000 But when I upload the same code to my Linux host, the output is 149. Why?
Positivity
  • 5,406
  • 6
  • 41
  • 61
2
votes
1 answer

I get problems regarding mathematical operations using an array php

I do mathematical calculations using two arrays like this $d_positif array(3) { [0]=> float(2.7742631687417) [1]=> float(2.5949809043991) [2]=> float(3.0174025996932) } $d_negatif array(3) { [0]=> float(2.7259998526469) [1]=>…
Bayu Angga Satrio
  • 131
  • 1
  • 11
2
votes
3 answers

How to round off a variable to n decimal places in c++

I have some float variables that would yield values like 1.23456789. I want to round it off to, say 4 decimal places. the setprecision function only roundoffs output, but I want to change the value of variable itself. So I'm looking for something…
Registered User
  • 2,239
  • 3
  • 32
  • 58
2
votes
1 answer

Why don't we have smaller float types in OpenGL and DirectX?

What's the problem with having lower than 10 bit floats? Why don't we have 8 bit floats? I can imagine how it would affect the outcome if glfloat is used for colors, but I can't imagine how it affects vertices. Do problems start to occur when we…
Uğur Gümüşhan
  • 2,455
  • 4
  • 34
  • 62
2
votes
1 answer

Incorrect plots using OpenGL renderer in MATLAB

I'm trying to plot a large amount of data on a graph using OpenGL. I want to use this renderer because it's the only one that supports transparency. The data is being plotted fine, but once you zoom in the lines connecting the points of data don't…
b3.
  • 7,094
  • 2
  • 33
  • 48
1 2 3
…
36 37