python3 can't run tokenize

Question

I run this program.Called [Keitaisokaiseki].

but I get the error

Traceback (most recent call last):
   File "C:\Users\Mao\Desktop\形態素解析　ランキング（栃木県）.py", line 23, in <module>
     tokens = t.tokenize(val)
   File "C:\Users\Mao\AppData\Local\Programs\Python\Python36-32\lib\site-
    packages\janome\tokenizer.py", line 194, in tokenize
    return list(self.__tokenize_stream(text, wakati, baseform_unk))
    File "C:\Users\Mao\AppData\Local\Programs\Python\Python36-32\lib\site-
    packages\janome\tokenizer.py", line 197, in __tokenize_stream
    text = text.strip()
   AttributeError: 'float' object has no attribute 'strip'

Line 194 is

    return list(self.__tokenize_stream(text, wakati, baseform_unk))

Line 197 is

    text = text.strip()

Give me any guidance.

can you please place your code in pastebin.com or some other codeshare place — Vasif, Dec 13 '17 at 06:41
What is `val`? Do the examples work? http://mocobeta.github.io/janome/en/ — OneCricketeer, Dec 13 '17 at 06:42
I mean.. OP linked a image so was thinking at least pastebin might be improvement. and also seems like an package issue due to possibly a wrong input. May be package mantainers will like it.. — Vasif, Dec 13 '17 at 06:43

score 0 · Answer 1 · answered Dec 13 '17 at 06:46

Looking at the code you provided you are trying to do

tokenize(val)

where val came from reading an excel sheet with xlrd. As can be seen here the return type from the xlrd call can be a float which explains your error as you are giving a float value to the tokenize function

Tomoko Uchida · Answer 2 · 2017-12-13T16:18:08.857

0

Seems you give float value to tokenize() method.

janome is a Japanese "text" analysis engine, so you need to cast numeric values to string before calling tokenize().

edited Dec 13 '17 at 16:18

answered Dec 13 '17 at 16:10

Tomoko Uchida

1
1

python3 can't run tokenize

2 Answers2