2

I'm attempting to run a basic character count using mrjob. The file is a unicode UTF-8 text document that contains, among other symbols, Chinese characters. When I run the character count, I only get counts of symbols in the ASCII character set returned.

As I understood it, mrjob worked with byte files and so should be able to process the unicode. Any idea how to make this work?

Character count code:

import collections
from mrjob.job import MRJob

import pdb

class MRCharCount(MRJob):
    def __init__(self, *args, **kwargs):
        super(MRCharCount, self).__init__(*args, **kwargs)
        self.charCount = collections.Counter()

    def mapper(self, _, value):
        #value = value.decode('utf-8')
        if 0:
            yield
        self.charCount += collections.Counter(value)

    def mapper_final(self):
        for (c, count) in self.charCount.iteritems():
            #c = c.decode('utf-8')
            #pdb.set_trace()
            yield c, count

    def reducer(self, key, values):
        #key = key.decode('utf-8')
        yield key, sum(values)

if __name__ == '__main__':
    MRCharCount.run()

Sample file:

"WantedCropped/13.jpg"  " 

蘑,-'此四郡咸瀉] 郡〝"同樹之吟[』赦〝連仙〝如五嶽〝Z蟲 總輝…埃 哪沛國樹也- 山嗚〕們義丘蜴〝紫陽鳳陽盎 枝也' 衡山泰山岱山也o委篤珊白俗因‵ 本踫順!〝 一抵以一 郡名而尊』冠〕，〝 夫[迥及砷洲咯l處嬤喵I奏 裔們彼此蕪汪一隃o4[姑聖叭}]五仰悔耐存蜘已吐琪〝 貴 天涯若北鄰 〝′ 聯盧吥以『千載重歡取硼」寫出子〝仃 『 之首) 苴〈垚思於此耳@ 仁 十] 本踫岫群屁〈屾}呱』力‵ 嚴〝'肋麻叭勘屾l支流喉巾清_I忖世鼻 { 次懸殊顯見舊本有歟. 征屾鮪以更) 例如'扣_『l屾 荳斗!閩之金釵溪踫喎、 慄沛國 郡文褶丕〞倣下十 屾







 】  【Iˋ 【】蔔lI_I(‵′l|忖|钀|ˋ| ˋ ˋ 〉_】IL 〝l



〝 咽,世揀侍皿^加〞剴]長予曰夢金懷 眸哪′博慶) .詛屾'戌… 材丕派下十世孫允石刀』懦寫予、 予_.趴更 正o

一】 蒼甫朱處派‵ 係友屾鞈忪派下廿四世孫公友 丕始遷) 其癆瞳鄢庸{父) 立以′公友一公 下五世蓀‵ 均求丕瀉肇基始淜) 紀以廿一 ll.lI屢」o 舊譜誤在公友雁者之上.啞闆荻〝'. 友一公是十』^世〝 稽穴』友蚕生於明嘉崝甲辰 即—扒』]兀一 五五如四年‵ 一世租 燃皿公居逕口』′ 是'公一兀「八七穴年‵ 其澗萋距有山〈百七十』^年

陡聶繭卯缸臟曹[囊 止喋 .寬芡一兀一 光冗 一_一」'曄__

'

Any ideas how Chinese character count could work? Thanks!

DevinRB
  • 107
  • 2
  • 8
  • The code in your example is incomplete. – ekhumoro Dec 06 '12 at 00:50
  • Code isn't incomplete, but I did have a typo. Corrected now. Thanks! – DevinRB Dec 06 '12 at 18:10
  • How do you open and read the file, and how is the data fed to the `MRCharCount` class. Also, why aren't you creating an instance of `MRCharCount`? – ekhumoro Dec 06 '12 at 18:26
  • On Ubuntu, run from the command line with python [-r emr] mrCharCount.py < mytextfile.txt > output.txt. Works for files with ASCII characters, works for ASCII characters (as a unicode subset) in the input file. – DevinRB Dec 07 '12 at 19:23

1 Answers1

1

It is not immediately obvious from the MRJob documentation exactly how MRJob interprets its input; it seems like .mapper() is being called with value set to a single line from the input file, as a byte-string. In that case, when you pass value to collections.Counter(), it will count the number of occurrences of each byte in the line, rather than each multi-byte character.

Therefore, I expect you should uncomment the line that says value = value.decode('utf-8'), and then it should work - or at least, be closer to working than before.

Screwtape
  • 429
  • 3
  • 11