NOTE: For this question I cannot use any imports other than io and sys
For an NLP assignment, I have to create a program that takes a grammar file and an utterance file as system arguments, which I've done.
The problem is, I'm so confused as to how to implement a deterministic CYK algorithm that outputs a string as an extended Chomsky Normal Form (eCNF).
I tried implementing a Node class, but I had a lot of trouble achieving the proper form. I also found implementations of probabilistic versions of the CYK algorithm, but they just confused me more. I don't want any probability scores.
I tried creating a matrix P[i][j] successfully, but it didn't come out as triangular and when I filled it with the words in my utterance line, it only took in the last word in the line.
This is the pseudocode I'm trying to follow:
Set P to a (n+1) x (n+1) matrix
for j = 1 to Length(words) do
for i = j-1 downto 0 do
for each non-terminal N in G do
P[i][j].NT[N] = empty array
for j = 1 to Length(words) do
for each rule N -> words[j] in G do
append (j, N -> words[j]) to P[j-1][j].NT[N]
change = true
while change do
change = false
for each non-terminal N in G do
if P[j-1][j].NT[N] is not empty and
there is a rule N' -> N in G and
(j, N' -> N) is not in P[j-1][j].NT[N'] then
append (j, N' -> N) to P[j-1][j].NT[N']
change = true
for j = 2 to Length(words) do
for i = j-2 downto 0 do
for k = i+1 to j-1 do
for each rule N -> A B in G such that
P[i][k].NT[A] is nonempty and
P[k][j].NT[B] is nonempty do
append (k, N -> A B) to P[i][j].NT[N]
change = true
while change do
change = false
for each non-terminal N in G do
if P[i][j].NT[N] is not empty and
there is a rule N' -> N in G and
(j, N' -> N) is not in P[i][j].NT[N'] then
append (j, N' -> N) to P[i][j].NT[N']
change = true
return P
Here are two example input files:
Grammar
S -> NP VP
NP -> Det N
NP -> PN
Det -> "the"
N -> "dog"
N -> "rat"
N -> "elephant"
PN -> "Alice"
PN -> "Bob"
VP -> V NP
V -> "admired"
V -> "bit"
V -> "chased"
Utterances
the aardvark bit the dog
the dog bit the man
Bob killed Alice
My program so far can tell when a sentence can be parsed and when it can't. Now I need to take utterances that can be parsed and parse them.
Output should look like the following:
[S [NP [Det "the"] [N "man"]] [VP [V "shot"] [NP [Det "the"] [N "elephant"]]]]
Here is my program with all the errors-inducing code removed:
import sys
import io
# usage = python CKYdet.py g#.ecfg u#L.utt
# Command Line Arguments - argv[0], argv[1], argv[2]
script = sys.argv[0]
grammarFile = open(sys.argv[1])
utteranceFile = open(sys.argv[2])
# Parsing algorithm
def CKYparse(uttline):
with open(sys.argv[1]) as rules:
# The following two lines throw index out of bound error. Not sure I need to select grammar rules this way.
# rhs = [line.split("-> ", 1)[1].strip('\n ') for line in rules]
# lhs = [line.split(None, 1)[0] for line in rules]
# Here I want to assign the words to their repective grammar rules
#Then I need to add each word to the matrix according to the grammar
#Then outpit the matrix with priper formatting
return "Valid parse goes here!" # Temporary return value until parse matrix P can be returned
# Initialize arrays from grammarFile
ruleArray = []
wordsInQuotes = []
for line in grammarFile:
rule = line.rstrip('\n')
start = line.find('"') + 1
end = line.find('"', start)
ruleArray.append(rule)
wordsInQuotes.append(line[start:end]) #create a set of words from grammar file
# Print final output
# Check whether line in utteranceFile can be parsed.
# If so, parse it.
# If not, print "No valid parse"
n = 0
for line in utteranceFile:
uttline = line
n = n + 1
uttString = "Utterance #{}: {}".format(n, line)
notValidString = "No valid parse\n"
if (all(x in wordsInQuotes for x in line.split())): #if word is found in grammarFile
print "".join((uttString, CKYparse(line)))
else:
print "".join((uttString, notValidString))
I understand the principles of the algorithm, but trying to write it in Python without NLTK is proving tricky.
