I am using whoosh package to do full-text fuzzy match.
My current code is as follows:
from whoosh.index import create_in
from whoosh.fields import *
from whoosh.query import FuzzyTerm
class MyFuzzyTerm(FuzzyTerm):
def __init__(self, fieldname, text, boost=1.0, maxdist=2, prefixlength=1, constantscore=True):
super(MyFuzzyTerm, self).__init__(fieldname, text, boost, maxdist, prefixlength, constantscore)
if not os.path.exists("indexdir"):
os.mkdir("indexdir")
path = u"MMM2.txt"
content = open('MMM2.txt', 'r').read()
schema = Schema(name=TEXT(stored=True), content=TEXT)
ix = create_in("indexdir", schema)
writer = ix.writer()
writer.add_document(name=path, content= content)
writer.commit()
from whoosh.qparser import QueryParser, FuzzyTermPlugin, PhrasePlugin, SequencePlugin
with ix.searcher() as searcher:
parser = QueryParser(u"content", ix.schema,termclass = MyFuzzyTerm)
parser.add_plugin(FuzzyTermPlugin())
parser.remove_plugin_class(PhrasePlugin)
parser.add_plugin(SequencePlugin())
str = u"Tennessee Riverkeeper Inc"
query = parser.parse(str)
# query = parser.parse(u"\"Tennessee Riverkeeper Inc\"~")
results = searcher.search(query)
print ("nb of results =", len(results),results, type(results))
for r in results:
print (r)
In document MMM2.txt, it contains the following text: "Tennessee aa Riverkeeper aa aa Inc". Ideally, I want the program returns 0 as I want to restrict the distance among the words in my term within 1. However, it still returns:
nb of results = 1 <Top 1 Results for And([MyFuzzyTerm('content', 'tennessee', boost=1.000000, maxdist=2, prefixlength=1), MyFuzzyTerm('content', 'riverkeeper', boost=1.000000, maxdist=2, prefixlength=1), MyFuzzyTerm('content', 'inc', boost=1.000000, maxdist=2, prefixlength=1)]) runtime=0.009658594451408662> <class 'whoosh.searching.Results'>
<Hit {'name': 'MMM2.txt'}>
However, if I replace:
query = parser.parse(str)
with:
query = parser.parse(u"\"Tennessee Riverkeeper Inc\"~")
It worked as I wanted to return non-matching results. I guess it has something to do with "~". But I cannot add it when I replace the string the variable name. Since I have so many strings to be matched, I cannot type them one by one. I can only store them into variables each time in the loop. Is there any way to solve this issue?
Thanks a lot for your help in advance!
