-2

I am writing a VSTi plugin, I need a coefficient calculating for part of the output envelope.

Basically I am receiving the incorrect answer to the mathematical expression, I have simplified it into several steps and it is still incorrect. The same expression works on a calculator, but it will not work translated into code. I have been using the debugger and checking the contents of the variables and have found that 0.25 / 88200 is giving a result of 2.8344671201814060e-006 and not 0.00000283446712 as it should be. Similar maths earlier in the code functions as expected.

currentDecCoeff = 0.25 / 88200;
is making currentDecCoeff = 2.8344671201814060e-006 (and not 0.00000283446712)

All relevant variables are of double type, I have even entered it as numeric values (as above) rather than variables, and it still gives the wrong answer.

Can anyone shed any light as to why this may be happening, and provide some ideas on things to try or even a solution.

I can add the code in context if needed. cheers

Ben

Nasty Fingers
  • 49
  • 6
  • The output is as expected: 0.0000028 == 2.8e-6 == 2.8*10^(-6) – harpun Jan 27 '13 at 18:48
  • Try it in Microsoft calculator and compare the answer. – Steve-o Jan 27 '13 at 18:48
  • 4
    Guys, no need to vote down this question. It's funny the OP never came across this notation, but where else to learn this then. May be a helpful post to many in the future. +1. – s.bandara Jan 27 '13 at 18:50
  • no never seen that notation before. 2.xxx leads me to believe the value is over 1 and not below as expected. Live and learn as the ethos of this website states. Thanks for your help anyway. – Nasty Fingers Jan 27 '13 at 18:54
  • Tried in MS calculator and result was in 0.00xxxetc. Additionally debugger highlighted the variable contents as out of range. – Nasty Fingers Jan 27 '13 at 19:00

2 Answers2

2

They are the same to 9 significant figures, look up scientific notation.

To elaborate, scientific notation is used to show very small (or very large) numbers easily by removing leading or trailing zeroes. Typically, it has the following form:

y x 10^z

To get the value, solve the sum. C shows this as:

yEz

Thus, 2.4x10^6 (or 2.4E6) is 2400000.

slugonamission
  • 9,562
  • 1
  • 34
  • 41
1

The notation e-006 means multiply what's left of me by 10^(-6). So indeed your result is correct.

s.bandara
  • 5,636
  • 1
  • 21
  • 36