Stack Overflow
Stack Overflow

Questions tagged [floating-point-precision]

Anything related to the precision of a floating-point number representation. The term precision refers to the number of significant digits a representation can hold. This is NOT the same as the "accuracy", which concerns errors in performing calculations, although it may be sometimes related.

551 questions
3
votes
1 answer

Extended floating point precision on mobile GPU

I'm trying to compute the gradient vector field of an image on the gpu using opengl-es 2.0. I found a cpu implementation for it which i use as a compare to my gpu implementation. The challenge here is that the cpu implementation relies on java type…
RogerLePatissier
  • 157
  • 11
3
votes
2 answers

iOS - remove trailing zeros from float WITHOUT converting to String?

I would like to remove trailing zeros from a float. I found multiple posts, but all answers suggest converting a float to NSString and using NSFormatter. While this might be good in most cases, I am performing XCTests and I need the values in float…
Fengson
  • 4,751
  • 8
  • 37
  • 62
3
votes
1 answer

"GL_HALF_FLOAT" with OpenGL Rendering and GLSL

I am programming an OpenGL renderer in C++. I want it to be as efficient as possible and each vertex/normal/UV tex coord/tangents/etc to take up as little memory as possible. I am using indexes, line strips, and fans. I was thinking that 32bit…
user3397315
  • 67
  • 1
  • 6
3
votes
1 answer

Trignometry in C# Cos & CosA

I am running into something that should be very simple to answer but i can't put my fingers on it. It has been quite sometime since i have done some trigonometry. double cosValue = -2.7105054312E-20; // (ACos) returns the angle var…
Sam
  • 875
  • 10
  • 22
3
votes
1 answer

Converting a long double to double with upward (or downward) rounding

Assume that we are working on a platform where the type long double has a strictly greater precision than 64 bits. What is the fastest way to convert a given long double to an ordinary double-precision number with some prescribed rounding (upward,…
iheap
  • 471
  • 5
  • 15
3
votes
1 answer

Java create BufferedImage with float precision

I created a map editor in Java. The problem is, I have steps for every byte value, so the map isn't smooth. Is it possible to change the BufferedImage raster data to float data and draw in float precision on it?
bitQUAKE
  • 473
  • 1
  • 8
  • 19
3
votes
1 answer

Should I combine multiplication and division steps when working with floating point values?

I am aware of the precision problems in floats and doubles, which why I am asking this: If I have a formula such as: (a/PI)*180.0 (where PI is a constant) Should I combine the division and multiplication, so I can use only one division:…
Rookie
  • 4,064
  • 6
  • 54
  • 86
3
votes
2 answers

General strategies for testing for equality of double precision numbers in computational geometry

So this seems to be a reoccurring problem for me- I'm trying to implement the line segment intersection and doubly connected edge list overlay algorithms in Computational Geometry by de Berg et al. Right now, I'm using the following function to test…
JustinBlaber
  • 4,629
  • 2
  • 36
  • 57
3
votes
2 answers

This very small float constant is more approximate than could be expected

public class Exponents { public static void main(String[] args) { float expFloat = 1.38e-43f; // 1.38 here System.out.println(expFloat); } } Output: 1.37E-43 This is crazy. The output should be 1.38 , other…
KASH J
  • 73
  • 5
3
votes
1 answer

Boole's rule for N intervals (C)

I am attempting to implement Boole's rule over n intervals using this formula So far I have developed this code: //f = function on the range [a,b] n = number of intervals long double booles(long double (*f) (long double), double…
CodeMonkey
  • 268
  • 5
  • 16
3
votes
1 answer

Changing Mantissa's Width in Non-IEEE Floating Point implementation

I have a gcc cross compiler on an 18 bit soft-core processor target that has the following datatypes defined: Integer 18 bit, Long 36 bit and float 36-bit(single precision). Right now my focus is on floating point operation. Since the width…
qzs
  • 31
  • 1
3
votes
0 answers

How to use @autojit with high precision floating point

I have been busy with my mendelbrot set generator program, which currently works with autojit (gpu parallelism) to increase performance. The thing is, it works only while floating-point is precise enough. Is there any way (or library) to increase…
Petar Mihalj
  • 51
  • 7
3
votes
1 answer

C floating point: Can x/x be > 1.0?

In C, can you ever exceed 1.0 if you have a number divided by itself? Basically, can x/x (e.g. 5.1/5.1) ever end up being greater than 1.0? x could be a float, double, or long double. Note that x/x wouldn't be literal code, like variable x over…
Ryan
  • 85
  • 6
3
votes
1 answer

Possible loss of precision with Gram-Schmidt

I have a code that uses Gram-Schmidt inside a loop. I want to reduce the number of calls to this algorithm as much as possible, but the thing is that despite of getting the same result before and after the call, when I print the results of some…
fc67
  • 409
  • 5
  • 17
3
votes
1 answer

What are the maximum number of values representable for different ranges within the floating-point representation?

I'm trying to figure out some specifics of using a normalized representation for floating-point values. As I understand, the IEEE-754 representation works like: 1.{significand} * 2^{exponent} For single precision, the significand being 24 bits, the…
Dan
  • 33,953
  • 24
  • 61
  • 87
1 2 3
…
36 37