3

Normally, I have a more technical problem but I will simplify it for you with an example of counting balls.

Assume I have balls of different colors and one index of an array (initialized to all 0's) reserved for each color. Every time I pick a ball, I increment the corresponding index by 1.

Balls are picked randomly and I can only pick one ball at a time. My sole purpose is to count number of balls for every color, until I run out of balls.

I would like to calculate standard deviation of the number of balls of different colors, while I am counting them. I do not want to calculate it by having to iterate through the array once more after I am done with counting all the balls.

To visualize:

Balls in random order: BBGRRYYBBGGGGGGB (each letter represents first letter of a color) Array indices from 0 to 3 correspond to colors B, G, R and Y respectively. When I am done picking the balls, my array looks like [5,7,2,2].

It is very simple to calculate standard deviation after having the final array but I want to do it while I am filling this array.

I want to do it in Java and I have approximately 1000 colors.

What is the most efficient way to implement that? Or is there even a way to do it before having the final array in hand?

David Nehme
  • 21,379
  • 8
  • 78
  • 117
Erol
  • 6,478
  • 5
  • 41
  • 55

2 Answers2

9

You don't need an array to calculate standard deviation.

Simply keep track of the number of points, total sum, and total sum of squares. You can calculate the mean and standard deviation at any time, without having to keep an array.

If I understand your requirements, you'll need a Map where the color is the key and an instance of Statistics is the value.

Here's a class that does it for you. 

package statistics;

/**
 * Statistics
 * @author Michael
 * @link http://stackoverflow.com/questions/11978667/online-algorithm-for-calculating-standrd-deviation/11978689#11978689
 * @since 8/15/12 7:34 PM
 */
public class Statistics {

    private int n;
    private double sum;
    private double sumsq;

    public void reset() {
        this.n = 0;
        this.sum = 0.0;
        this.sumsq = 0.0;
    }

    public synchronized void addValue(double x) {
        ++this.n;
        this.sum += x;
        this.sumsq += x*x;
    }

    public synchronized double calculateMean() {
        double mean = 0.0;
        if (this.n > 0) {
            mean = this.sum/this.n;
        }
        return mean;
    }

    public synchronized double calculateVariance() {
       double deviation = calculateStandardDeviation();
        return deviation*deviation;
    }

    public synchronized double calculateStandardDeviation() {
        double deviation = 0.0;
        if (this.n > 1) {
            deviation = Math.sqrt((this.sumsq - this.sum*this.sum/this.n)/(this.n-1));
        }
        return deviation;
    }
}

Here is its unit test:

package statistics;

import org.junit.Assert;
import org.junit.Test;

/**
 * StatisticsTest
 * @author Michael
 * @link http://www.wolframalpha.com/input/?i=variance%281%2C+2%2C+3%2C+4%2C+5%2C+6%29&a=*C.variance-_*Variance-
 * @since 8/15/12 7:42 PM
 */
public class StatisticsTest {

    private static final double TOLERANCE = 1.0E-9;

    @Test
    public void testCalculateMean() {
        double [] values = new double[] {
            1.0, 2.0, 3.0, 4.0, 5.0, 6.0
        };
        Statistics stats = new Statistics();
        for (double value : values) {
            stats.addValue(value);
        }
        double expected = 3.5;
        Assert.assertEquals(expected, stats.calculateMean(), TOLERANCE);
    }

    @Test
    public void testCalculateVariance() {
        double [] values = new double[] {
                1.0, 2.0, 3.0, 4.0, 5.0, 6.0
        };
        Statistics stats = new Statistics();
        for (double value : values) {
            stats.addValue(value);
        }
        double expected = 3.5;
        Assert.assertEquals(expected, stats.calculateVariance(), TOLERANCE);
    }


    @Test
    public void testCalculateStandardDeviation() {
        double [] values = new double[] {
                1.0, 2.0, 3.0, 4.0, 5.0, 6.0
        };
        Statistics stats = new Statistics();
        for (double value : values) {
            stats.addValue(value);
        }
        double expected = Math.sqrt(3.5);
        Assert.assertEquals(expected, stats.calculateStandardDeviation(), TOLERANCE);
    }

}
duffymo
  • 305,152
  • 44
  • 369
  • 561
  • How can I keep track of total sum of squares without knowing how many balls I have for each color? – Erol Aug 15 '12 at 23:26
  • 2
    You need an array to keep track of the count of each color. However, you don't need to recalculate the avg and stddev from scratch. Since you know the color of the current ball, you can subtract out it's previous square then add in the new square. – Code-Apprentice Aug 15 '12 at 23:30
  • 1
    @duffymo I don't think your answer is complete. Part of the problem is that the OP needs to *change* a previous value when the next input is read. – Code-Apprentice Aug 16 '12 at 00:19
  • @Code-Guru, that's totally correct. I don't have the array ready when I am calculating standard deviation. That's the core of my problem and this answer does not address that. – Erol Aug 16 '12 at 00:22
1

Since average and standard deviation are calculated using sums, you can easily implement appropriate accumulators for these. Then when you want the actual values, finish the rest of the calculation (particularly, the divisions).

The sum of squares is the tricky part since you increment one of the frequencies for each input. One way to deal with this is to maintain a count of each color seen so far (using an appropriate data structure). Then when you see a color in the input, you can subtract out its previous square and add the new square back in (or equivalently add the difference of the two squares to your accumulator).

I'll leave it to the reader to implement the algorithm described here.

Code-Apprentice
  • 81,660
  • 23
  • 145
  • 268