0

Consider a file containing words separated by spaces; write a MapReduce program in Python, which counts the number of times each 3-word sequence appears in the file.

For example, consider the following file:

one two three seven one two three
three seven one
seven one two

The number of times each 3 word sequence appears in this file is:

"three seven one" 2
"four seven one two" 1
"one two three" 2
"seven one two" 2
"two three seven" 1

Code format:

from mrjob.job import MRJob


class MR3Nums(MRJob):
    
    def mapper(self,_, line):
        pass

    def reducer(self,key, values):
        pass
    

if __name__ == "__main__":
    MR3Nums.run()
John Whitehouse
  • 3
  • 2

1 Answers1

0

The mapper is applied on each line, and should count each 3-word sequence, i.e. yield the 3-word sequence along with a count of 1.

The reducer is called with key and values, where key is a 3-word sequence and values is a list of counts (which would be a list of 1s). The reducer can simply return a tuple of the 3-word sequence and the total number of occurrences, the latter obtained via sum.

class MR3Nums(MRJob):
    
    def mapper(self, _, line):
        sequence_length = 3
        words = line.strip().split()
        for i in range(len(words) - sequence_length + 1):
            yield " ".join(words[i:(i+sequence_length)]), 1

    def reducer(self, key, values):
        yield key, sum(values)
Anson Miu
  • 1,171
  • 7
  • 6