how to get numbers to have precision of .05?

Question

The following will ensure that any large numbers will only be precise to the hundredths place (related to this answer):

public function round( sc:Number ):Number
{
    sc = sc * 100;
    sc = Math.floor( sc );
    sc = sc / 100;

    return sc;
}

What is the optimal way to round my numbers to the precision of .05? Is there something clever to be done with bit-shifting to get this result? For example, I would like:

3.4566 = 3.45

3.04232 = 3.05

3.09 = 3.1

3.32 = 3.3

Chris Thornhill · Accepted Answer · 2009-04-28T20:12:52.033

6

You could multiply by 20, round, then divide by 20.

Edit: You will want to use Math.round() instead of Math.floor(), otherwise your test case 3.09 will turn into 3.05.

edited Apr 28 '09 at 20:12

answered Apr 28 '09 at 20:07

Chris Thornhill

5,321
1
24
21

score 2 · Answer 2 · answered Apr 28 '09 at 20:06

2

You could just change the *100 to *20 and /100 /20. But why stop there, when you can naturally generalize?

double floor(double in, double precision) {
    return Math.floor(in/precision)*precision
}
//floor(1.07, 0.05) = 1.05

answered Apr 28 '09 at 20:06

Craig Gidney

17,763
5
68
136

This should be Math.round(), not Math.floor(). – Svante Apr 28 '09 at 21:02
His example used floor, so I used floor. Also notice I called the function floor to make it clear. – Craig Gidney Apr 28 '09 at 22:45
To get consistent results for negative numbers you should also add 0.5 to the "in / precision" result passed to Math.floor() – Alnitak Apr 29 '09 at 13:24
That would make it a rounding function, not a floor function. Floor rounds towards negative infinity. What you're thinking of is called truncation. – Craig Gidney Apr 29 '09 at 14:33

how to get numbers to have precision of .05?

2 Answers2