Get N samples given iterator

Question

Given are an iterator it over data points, the number of data points we have n, and the maximum number of samples we want to use to do some calculations (maxSamples).

Imagine a function calculateStatistics(Iterator it, int n, int maxSamples). This function should use the iterator to retrieve the data and do some (heavy) calculations on the data element retrieved.

• if n <= maxSamples we will of course use each element we get from the iterator
• if n > maxSamples we will have to choose which elements to look at and which to skip

I've been spending quite some time on this. The problem is of course how to choose when to skip an element and when to keep it. My approaches so far:

• I don't want to take the first maxSamples coming from the iterator, because the values might not be evenly distributed.
• Another idea was to use a random number generator and let me create maxSamples (distinct) random numbers between 0 and n and take the elements at these positions. But if e.g. n = 101 and maxSamples = 100 it gets more and more difficult to find a new distinct number not yet in the list, loosing lot of time just in the random number generation
• My last idea was to do the contrary: to generate n - maxSamples random numbers and exclude the data elements at these positions elements. But this also doesn't seem to be a very good solution.

Do you have a good idea for this problem? Are there maybe standard known algorithms for this?

Answer 1

To provide some answer, a good way to collect a set of random numbers given collection size > elements needed, is the following. (in C++ ish pseudo code).

EDIT: you may need to iterate over and create the "someElements" vector first. If your elements are large they can be "pointers" to these elements to save space.

vector randomCollectionFromVector(someElements, numElementsToGrab) {
    while(numElementsToGrab--) {
         randPosition = rand() % someElements.size();
         resultVector.push(someElements.get(randPosition))
         someElements.remove(randPosition);
    }
    return resultVector;
}

If you don't care about changing your vector of elements, you could also remove random elements from someElements, as you mentioned. The algorithm would look very similar, and again, this is conceptually the same idea, you just pass someElements by reference, and manipulate it.

Something worth noting, is the quality of psuedo random distributions as far as how random they are, grows as the size of the distribution you used increases. So, you may tend to get better results if you pick which method you use based on which method results in the use of more random numbers. Example: if you have 100 values, and need 99, you should probably pick 99 values, as this will result in you using 99 pseudo random numbers, instead of just 1. Conversely, if you have 1000 values, and need 99, you should probably prefer the version where you remove 901 values, because you use more numbers from the psuedo random distribution. If what you want is a solid random distribution, this is a very simple optimization, that will greatly increase the quality of "fake randomness" that you see. Alternatively, if performance matters more than distribution, you would take the alternative or even just grab the first 99 values approach.

Answer 2

interval = n/(n-maxSamples) //an euclidian division of course
offset = random(0..(n-1)) //a random number between 0 and n-1
totalSkip = 0
indexSample = 0;
FOR it IN samples DO
    indexSample++ // goes from 1 to n
    IF totalSkip < (n-maxSamples) AND indexSample+offset % interval == 0 THEN
        //do nothing with this sample
        totalSkip++
    ELSE
        //work with this sample
    ENDIF
ENDFOR
ASSERT(totalSkip == n-maxSamples) //to be sure

interval represents the distance between two samples to skip. offset is not mandatory but it allows to have a very little diversity.

Answer 3

Based on the discussion, and greater understanding of your problem, I suggest the following. You can take advantage of a property of prime numbers that I think will net you a very good solution, that will appear to grab pseudo random numbers. It is illustrated in the following code.

#include <iostream>
using namespace std;


int main() {
    const int SOME_LARGE_PRIME = 577;  //This prime should be larger than the size of your data set.  
    const int NUM_ELEMENTS = 100;
    int lastValue = 0;
    for(int i = 0; i < NUM_ELEMENTS; i++) {
        lastValue += SOME_LARGE_PRIME;
        cout << lastValue % NUM_ELEMENTS << endl;
    }
}

Using the logic presented here, you can create a table of all values from 1 to "NUM_ELEMENTS". Because of the properties of prime numbers, you will not get any duplicates until you rotate all the way around back to the size of your data set. If you then take the first "NUM_SAMPLES" of these, and sort them, you can iterate through your data structure, and grab a pseudo random distribution of numbers(not very good random, but more random than a pre-determined interval), without extra space and only one pass over your data. Better yet, you can change the layout of the distribution by grabbing a random prime number each time, again must be larger than your data set, or the following example breaks.

PRIME = 3, data set size = 99. Won't work.

Of course, ultimately this is very similar to the pre-determined interval, but it inserts a level of randomness that you do not get by simply grabbing every "size/num_samples"th element.

Answer 4

This is called the Reservoir sampling