Why is a list of cumulative frequency sums required for implementing a random word generator?

Question

I'm working on exercise 13.7 from Think Python: How to Think Like a Computer Scientist. The goal of the exercise is to come up with a relatively efficient algorithm that returns a random word from a file of words (let's say a novel), where the probability of the word being returned is correlated to its frequency in the file.

The author suggests the following steps (there may be a better solution, but this is assumably the best solution for what we've covered so far in the book).

1. Create a histogram showing {word: frequency}.
2. Use the keys method to get a list of words in the book.
3. Build a list that contains the cumulative sum of the word frequencies, so that the last item in this list is the total number of words in the book, n.
4. Choose a random number from 1 to n.
5. Use a bisection search to find the index where the random number would be inserted in the cumulative sum.
6. Use the index to find the corresponding word in the word list.

My question is this: What's wrong with the following solution?

1. Turn the novel into a list t of words, exactly as they as they appear in the novel, without eliminating repeat instances or shuffling.
2. Generate a random integer from 0 to n, where n = len(t) – 1.
3. Use that random integer as an index to retrieve a random word from t.

Thanks.

Answer 1

Your approach is (also) correct, but it uses space proportional to the input text size. The approach suggested by the book uses space proportional only to the number of distinct words in the input text, which is usually much smaller. (Think about how often words like "the" appear in English text.)