counting relative frequency in pairs a strips mapreduce

Question

i am new in python and i want to use MrJob package for countind relative frequency of pair words i wrote below code but it doesn't make correct output. can you plz help me with my mistakes? (|) = (, )/()=(, )/∑A' (′ , )

import re
from collections import defaultdict

from mrjob.job import MRJob

WORD_RE = re.compile(r"[\w']+")


class MRRelativeFreq(MRJob):
    def mapper(self, _, line):
        for word in WORD_RE.findall(line):
          for wordpair in WORD_RE.findall(line):
            if word != wordpair:
               yield (word.lower(), wordpair.lower(), 1)

    def reducer(self, key, values):
        cnts = defaultdict(int)
        total = 0
        for (word, count) in values:
          cnt=0
          total += count
          cnts[word] += count

        for (k,kp), v in cnts.items():
            yield (k,kp), (v, float(v) / total) 

    def combiner(self, key, values):
        yield None, (key, sum(values))


if __name__ == '__main__':
    MRRelativeFreq.run()

your code does not make sense, why do you use `findall` twice? also why not use the code shown here --> https://mrjob.readthedocs.io/en/latest/guides/quickstart.html#writing-your-first-job — gold_cy, Dec 19 '21 at 13:24
because it just return the frequency of each terms alone, I want to return relative frequency of terms with pairs method — Learner, Dec 19 '21 at 13:28
can you edit the question to include a sample desired output — gold_cy, Dec 19 '21 at 13:32
Let us [continue this discussion in chat](https://chat.stackoverflow.com/rooms/240266/discussion-between-gold-cy-and-learner). — gold_cy, Dec 19 '21 at 13:40
Does the word compile should be: `WORD_RE = re.compile(r"[a-zA-Z'-]+")` — Waket Zheng, Dec 19 '21 at 14:20

gold_cy · Accepted Answer · 2021-12-19T21:44:47.937

1

You will need an intermediate data structure, in this case a defaultdict to count the total of times the word appears.

import re
from collections import defaultdict
from itertools import combinations

from mrjob.job import MRJob
from mrjob.step import MRStep

WORD_RE = re.compile(r"[\w']+")


class MRRelativeFreq(MRJob):
    denoms = defaultdict(int)

    def steps(self):
        return [
            MRStep(
                mapper=self.mapper,
                combiner=self.combiner,
                reducer=self.reducer),
            MRStep(
                reducer=self.reducer_s2)
        ]

    def mapper(self, _, line):
        words = WORD_RE.findall(line)
        for (x, y) in combinations(words, 2):
            if x != y:
                yield ((x.lower(), "*"), 1)
                yield ((x.lower(), y.lower()), 1)

    def combiner(self, pair, counts):
        yield (pair, sum(counts))

    def reducer(self, pair, counts):
        count = sum(counts)
        x, y = pair
        if y == "*":
            self.denoms[x] = count
        else:
            yield ((x, y), count)

    def reducer_s2(self, pair, ycnt):
        x, y = pair
        lkup = self.denoms[x]
        yield (pair, round((sum(ycnt) / lkup), 2))

if __name__ == '__main__':
    MRRelativeFreq.run()

Given a file of the below structure:

"""
this is something
this is not
or else this is
"""

I get the following results:

["or", "else"]  0.33
["or", "is"]    0.33
["or", "this"]  0.33
["this", "is"]  0.6
["this", "not"] 0.2
["this", "something"]   0.2
["is", "not"]   0.5
["is", "something"] 0.5
["else", "is"]  0.5
["else", "this"]    0.5

Used this as a hint.

edited Dec 19 '21 at 21:44

answered Dec 19 '21 at 14:24

gold_cy

13,648
3
23
45

great answer thanks, i found from output that, this code only pair each term with terms after itself not considering the term before it – Learner Dec 20 '21 at 09:27
does this solution answer your question? – gold_cy Dec 20 '21 at 12:04
not really, i changed somewhere of this code and find how to pair and count pairs but it is not my answer. i share my code in chatbox – Learner Dec 20 '21 at 12:11
the code in the chat is literally the same code I posted – gold_cy Dec 20 '21 at 12:44
the mapper is a bit different. your code help me a lot thank – Learner Dec 20 '21 at 12:51
is it correct to have 100 pair in output by this code? def reducer_cout(self, _, relfreq): for i in range(100): yield max(relfreq) – Learner Dec 20 '21 at 12:54
at this point I don't understand what you are asking anymore. I am sorry but I don't think I can help you any further. you are asking additional questions within this one. – gold_cy Dec 20 '21 at 12:57

counting relative frequency in pairs a strips mapreduce

1 Answers1