0

Using Sybase ASE 12.5 I have the following situation.

2 values stored in float cols when multiplied give a value.

Converting that value to a varchar (or retrieving it with Java) gives the underlying precise value which the floats approximated to.

My issue is that the value as represented by the floats is correct, but the precise value is causing issues (due to strict rounding rules).

For example

declare @a float,@b float
select @a = 4.047000, @b = 1033000.000000
select @a*@b as correct , str(@a*@b,40,20) as wrong

gives:

correct: 4180551.000000,
wrong: 4180550.9999999995343387

Similarly when @a = 4.047000, @b = 1 ...you get 

correct: 4.047000,
wrong: 4.0469999999999997

(same thing happens using convert(varchar(30), @a*@b) and cast(@a*@b, varchar(30) )

I appreciate it would be easy to just round the first example in java but for various business reasons that cannot be done and in any case it wouldn't work for the second.

I also cannot change the float table column datatype.

Is there anyway to get the float representation of the multiplication product either as a string or the actual 'correct' value above?

Thanks Chris

Danny Beckett
  • 20,529
  • 24
  • 107
  • 134
GlorianChris
  • 1
  • 1
  • I can't answer for Sybase, but you've got 23 decimal digits out of a value that can hold 16-17 decimal digits. When you do this, you get nonsense for the extra digits. It's a tricky area of computing (for exhaustive details, search for 'What every computer scientist should know about floating point arithmetic', widely quoted on SO and easily findable on the web). The best thing is to limit the output to 16 decimal digits in total. – Jonathan Leffler Jan 10 '13 at 17:23
  • What I really need is the value as displayed in the 'correct' value above regardless of the underlying accurate value. If I can display the correct value I should be able to store it somehow right? – GlorianChris Jan 10 '13 at 17:25

0 Answers0