Can't get my head around why standard Python code produces an unexpected result when translated to MapReduce using mrjob.
Example data from a .txt file:
1 12
1 14
1 15
1 16
1 18
1 12
2 11
2 11
2 13
3 12
3 15
3 11
3 10
This code creates a dictionary and performs a simple division calculation:
dic = {}
with open('numbers.txt', 'r') as fi:
for line in fi:
parts = line.split()
dic.setdefault(parts[0],[]).append(int(parts[1]))
print(dic)
for k, v in dic.items():
print (k, 1/len(v), v)
Result:
{'1': [12, 14, 15, 16, 18, 12], '2': [11, 11, 13], '3': [12, 15, 11, 10]}
1 0.16666666666666666 [12, 14, 15, 16, 18, 12]
2 0.3333333333333333 [11, 11, 13]
3 0.25 [12, 15, 11, 10]
But when translated to MapReduce using mrjob:
from mrjob.job import MRJob
from mrjob.step import MRStep
from collections import defaultdict
class test(MRJob):
def steps(self):
return [MRStep(mapper=self.divided_vals)]
def divided_vals(self, _, line):
dic = {}
parts = line.split()
dic.setdefault(parts[0],[]).append(int(parts[1]))
for k, v in dic.items():
yield (k, 1/len(v)), v
if __name__ == '__main__':
test.run()
Result:
["2", 1.0] [11]
["2", 1.0] [13]
["3", 1.0] [12]
["3", 1.0] [15]
["3", 1.0] [11]
["3", 1.0] [10]
["1", 1.0] [12]
["1", 1.0] [14]
["1", 1.0] [15]
["1", 1.0] [16]
["1", 1.0] [18]
["1", 1.0] [12]
["2", 1.0] [11]
Why doesn't MapReduce group and calculate in the same way? How do I recreate the standard Python result in MapReduce?
