2

I am working on Language Modeling using nltk I am using this essay as my corpus in mypet.txt file. I am getting 0.25 Kneser Ney probability distribution for most of the trigrams. I don't know why. Is it right? Why is it doing so? This is my word_ngram.py file:

import io
import nltk
from nltk.util import ngrams
from nltk.tokenize import sent_tokenize
from preprocessor import utf8_to_ascii

with io.open("mypet.txt",'r',encoding='utf8') as utf_file:
    file_content = utf_file.read()

ascii_content = utf8_to_ascii(file_content)
sentence_tokenize_list = sent_tokenize(ascii_content)

all_tgrams = []
for sentence in sentence_tokenize_list:
    sentence = sentence.rstrip('.!?')
    tokens = nltk.re.findall(r"\w+(?:[-']\w+)*|'|[-.(]+|\S\w*", sentence)
    tgrams = ngrams(tokens, 3,pad_left=True,pad_right=True,left_pad_symbol='<s>', right_pad_symbol="</s>")
    all_tgrams.extend(tgrams)

frequency_distribution = nltk.FreqDist(all_tgrams)
kneser_ney = nltk.KneserNeyProbDist(frequency_distribution)
for i in kneser_ney.samples():
    print "{0}: {1}".format(kneser_ney.prob(i), i)

This is my preprocessor.py file which is handling utf-8 chars:

# -*- coding: utf-8 -*-

import json


def utf8_to_ascii(utf8_text):
    with open("utf_to_ascii.json") as data_file:
        data = json.load(data_file)
    utf_table = data["chars"]
    for key, value in utf_table.items():
        utf8_text = utf8_text.replace(key, value)
    return utf8_text.encode('ascii')

And this is my utf_to_ascii.json file which I used to replace utf-8 char to ascii char:

{
 "chars": {
          "“":"",
          "”":"",
          "’":"'",
          "—":"-",
          "–":"-"
 }
}

This is sample output of few trigrams:

0.25: ('side', '</s>', '</s>')
0.25: ('I', 'throw', 'a')
0.25: ('it', 'to', 'us')
0.25: ('guards', 'the', 'house')
0.0277777777778: ('<s>', 'I', 'am')
0.25: ('a', 'fire', 'broke')
0.125: ('our', 'house', 'at')
0.25: ('that', 'a', 'heap')
0.25: ('is', 'covered', 'with')
0.25: ('with', 'a', 'soft')
0.00862068965517: ('<s>', 'It', 'begins')
0.25: ('swim', '</s>', '</s>')
0.25: ('a', 'member', 'of')
0.25: ('bread', '</s>', '</s>')
0.25: ('love', '</s>', '</s>')
0.25: ('a', 'soft', 'fur')
0.25: ('body', 'is', 'covered')
0.25: ('I', 'bathe', 'it')
0.25: ('it', 'is', 'out')
0.25: ('<s>', 'A', 'thief')
0.25: ('go', 'hunting', '</s>')
0.025: ('It', 'is', 'loved')
0.25: ('it', 'a', 'loving')
0.25: ('with', 'soap', 'every-day')
0.25: ('other', 'members', 'of')
0.25: ('lying', 'there', 'was')
0.25: ('sensitive', 'to', 'sound')
0.25: ('and', 'the', 'flames')
0.25: ('kitchen', '</s>', '</s>')
0.25: ('strong', 'instinct', '</s>')
Jai Prak
  • 2,855
  • 4
  • 29
  • 37

1 Answers1

2

Probably by this time you have figured it out already... Anyway I would use nltk's collocations instead.

You need to get your list of words more or less the same way:

all_words = []
for sentence in sentence_tokenize_list:
    sentence = sentence.rstrip('.!?')
    tokens = nltk.re.findall(r"\w+(?:[-']\w+)*|'|[-.(]+|\S\w*", sentence)
    all_words.append('<s>')
    all_words.extend(tokens)
    all_words.append('</s>')
all_words[:9]

['<s>', 'I', 'am', 'fond', 'of', 'pets', '</s>', '<s>', 'I']

Then you create your collocations finder:

from nltk.collocations import TrigramCollocationFinder, TrigramAssocMeasures
tcf = TrigramCollocationFinder.from_words(all_words)
trigram_measures = TrigramAssocMeasures()

The following would give you the best 10 trigrams according to frequency:

tcf.nbest(trigram_measures.raw_freq, 10)
[('</s>', '<s>', 'It'),
 ('<s>', 'It', 'is'),
 ('</s>', '<s>', 'I'),
 ('<s>', 'It', 'runs'),
 ('house', '</s>', '<s>'),
 ('</s>', '<s>', 'At'),
 ('It', 'is', 'very'),
 ('It', 'runs', 'after'),
 ('it', '</s>', '<s>'),
 ('the', 'house', '</s>')]

You would usually want to filter some of them, e.g. the less frequent, although your dataset is small.

tcf.apply_freq_filter(2)
tcf.apply_ngram_filter(lambda w1, w2, w3: ("<s>" in [w1, w2, w3]) and ("</s>" in [w1, w2, w3]))

tcf.nbest(trigram_measures.raw_freq, 10)

[('<s>', 'It', 'is'),
 ('<s>', 'It', 'runs'),
 ('It', 'is', 'very'),
 ('It', 'runs', 'after'),
 ('the', 'house', '</s>'),
 ('<s>', 'I', 'have'),
 ('<s>', 'It', 'has'),
 ('It', 'has', 'a'),
 ('member', 'of', 'the'),
 ('of', 'the', 'house')]

Finally you apply the Kneser Ney smoothing.

kneser_ney = nltk.KneserNeyProbDist(tcf.ngram_fd)
for i in tcf.nbest(trigram_measures.raw_freq, 10):
    print("{0}: {1}".format(kneser_ney.prob(i), i))

0.31896551724137934: ('<s>', 'It', 'is')
0.11206896551724138: ('<s>', 'It', 'runs')
0.225: ('It', 'is', 'very')
0.5625: ('It', 'runs', 'after')
0.5625: ('the', 'house', '</s>')
0.1388888888888889: ('<s>', 'I', 'have')
0.04310344827586207: ('<s>', 'It', 'has')
0.625: ('It', 'has', 'a')
0.625: ('member', 'of', 'the')
0.3125: ('of', 'the', 'house')

Note that these are the original frequencies if you just used the trigram.

for i in tcf.nbest(trigram_measures.raw_freq, 10):
    print("{0}: {1}".format(tcf.score_ngram(trigram_measures.raw_freq, *i), i))

0.019083969465648856: ('<s>', 'It', 'is')
0.007633587786259542: ('<s>', 'It', 'runs')
0.0057251908396946565: ('It', 'is', 'very')
0.0057251908396946565: ('It', 'runs', 'after')
0.0057251908396946565: ('the', 'house', '</s>')
0.003816793893129771: ('<s>', 'I', 'have')
0.003816793893129771: ('<s>', 'It', 'has')
0.003816793893129771: ('It', 'has', 'a')
0.003816793893129771: ('member', 'of', 'the')
0.003816793893129771: ('of', 'the', 'house')
lrnzcig
  • 3,868
  • 4
  • 36
  • 50