I have read the text1.similar("monstrous") and text1.concordance("monstrous") from this.
Where I couldn't get the satisfactory answer for the difference between text1.concordance('monstrous') and text1.similar('monstrous') of natural language processing toolkit in python.
So would you please give the explanation with an example in detail?
Using concordance(token) gives you the context surrounding the argument token. It will show you the sentences where token appears.
Using similar(token) returns a list of words that appear in the same context as token. In this case the the context is just the words directly on either side of token.
So, looking at the Moby Dick text (text1). We can check the concordance of 'monstrous'
text1.concordance('monstrous')
# returns:
Displaying 11 of 11 matches:
ong the former , one was of a most monstrous size . ... This came towards us ,
ON OF THE PSALMS . " Touching that monstrous bulk of the whale or ork we have r
ll over with a heathenish array of monstrous clubs and spears . Some were thick
d as you gazed , and wondered what monstrous cannibal and savage could ever hav
that has survived the flood ; most monstrous and most mountainous ! That Himmal
they might scout at Moby Dick as a monstrous fable , or still worse and more de
th of Radney .'" CHAPTER 55 Of the Monstrous Pictures of Whales . I shall ere l
ing Scenes . In connexion with the monstrous pictures of whales , I am strongly
ere to enter upon those still more monstrous stories of them which are to be fo
ght have been rummaged out of this monstrous cabinet there is no telling . But
of Whale - Bones ; for Whales of a monstrous size are oftentimes cast up dead u
And then we can get a list of words that appear in similar contexts to 'monstrous'. The context for the first returned line is 'most _____ size'.
text1.similar('monstrous')
# returns:
determined maddens contemptible modifies abundant tyrannical puzzled
trustworthy impalpable gamesome curious mean pitiable untoward
christian subtly passing domineering uncommon true
If we take the word 'true' and check it's concordance with text.concordance('true') we will get back the first 25 of 87 uses of the word 'true'. This isn't terribly useful, but NLTK does provide an additional method called common_contexts that shows when the use of a list of words share the same surrounding words.
text1.common_contexts(['monstrous', 'true'])
# returns:
the_pictures
This result tells us that the phrases "the monstrous pictures" and "the true pictures" both appear in Moby Dick.
I will explain with example:
text1.similar("monstrous")
will output the words with similar context such as word1 ______ word2. For example it outputs the word doleful. If you run:
text1.concordance("monstrous")
You will see among the matches the line:
that has survived the flood ; most monstrous and most mountainous ! That Himmal
If you run:
text1.concordance("doleful")
You will see among the matches the line:
ite perspectives . There ' s a most doleful and most mocking funeral ! The sea
And
text1.common_contexts(["monstrous", "doleful"])
will output common surrounding words of monstrous and doleful which are "most" and "and"
most_and
Concordance(token) provides you with the context in which a token is used.
Similar(token) provides you with other words that appear in similar contexts.
To illustrate, here a more general description to approximate their functionality.
1) Concordance(token):
This returns a predefined number of words to the left and right of your token (let's call this collection of words "Z"). It does that for each instance that your token appears in the text.
2) similar(token):
A word will be listed here if its occurrence within the words of the set "Z" is rather likely.
I will explain with an exaustive example of the word "monstrous" and its similars on the book Sense and Sensibility by Jane Austen 1811 (text2):
common_contexts shared by "monstrous" and "very": ['a_pretty', 'am_glad', 'a_lucky', 'is_pretty', 'be_glad']
context: a_pretty
"a very pretty"
ct . She was short and plump , had a very pretty face , and the finest express
some delightful stew - ponds , and a very pretty canal ; and every thing , in
from Norland , Elinor had painted a very pretty pair of screens for her siste
"a monstrous pretty"
. " Now , Palmer , you shall see a monstrous pretty girl ." He immediately went
context: am_glad
"am very glad"
ouble you with my affairs . But I am very glad to find it was only my own fanc
be ordained in readiness ; and I am very glad to find things are so forward b
"am monstrous glad"
your sister is to marry him . I am monstrous glad of it , for then I shall have
usual noisy cheerfulness , " I am monstrous glad to see you -- sorry I could n
t however , as it turns out , I am monstrous glad there was never any thing in
possible that she should ." " I am monstrous glad of it . Good gracious ! I hav
context: a_lucky
"a very lucky"
Colonel Brandon may be , Edward is a very lucky man .-- You will not mention t
"a monstrous lucky"
ou may tell your sister . She is a monstrous lucky girl to get him , upon my ho
context: is_pretty
"is very pretty"
t is only the Palmers . Charlotte is very pretty , I can tell you . You may se
" All this ," thought Elinor , " is very pretty ; but it can impose upon neit
eau of mine ." " Aye , aye , that is very pretty talking -- but it won ' t do
"is monstrous pretty"
k how you will like them . Lucy is monstrous pretty , and so good humoured and
context: be_glad
"be very glad"
trusting YOU ; indeed , I should be very glad of your advice how to manage in
o ever he might have , she should be very glad to have it all , you know , or
ant company , I am sure we should be very glad to come and stay with her for a
y ' s sister , my dear . I should be very glad to get her so good a mistress .
patience !-- And though one would be very glad to do a kindness by poor Mr . F
d , assuring her that Fanny would be very glad to see her , invited her to com
"be monstrous glad"
Jennings , " I am sure I shall be monstrous glad of Miss Marianne ' s company
common_contexts shared by "monstrous" and "so": ['am_glad', 'is_pretty']
context: am_glad
"am so glad"
delight in seeing them again . " I am so glad to see you !" said she , seating
eems very agreeable ." " Well -- I am so glad you do . I thought you would , h
small , was pleasing to her . " I am so glad we are got acquainted at last ,"
an be like it , to be sure ! And I am so glad your sister is going to be well
d otherwise have been learnt . " I am so glad to meet you ;" said Miss Steele
"am monstrous glad"
your sister is to marry him . I am monstrous glad of it , for then I shall have
usual noisy cheerfulness , " I am monstrous glad to see you -- sorry I could n
t however , as it turns out , I am monstrous glad there was never any thing in
possible that she should ." " I am monstrous glad of it . Good gracious ! I hav
context: is_pretty
"is so pretty"
n it must be some other place that is so pretty I suppose ." When they were se
"is monstrous pretty"
k how you will like them . Lucy is monstrous pretty , and so good humoured and
common_contexts shared by "monstrous" and "exceedingly": ['be_glad', 'is_fond']
context: be_glad
"be exceedingly glad"
but , my dear Elinor , I shall be exceedingly glad to know more of it ; and I t
with her ; otherwise I should be exceedingly glad to do it . You know I am alw
"be monstrous glad"
Jennings , " I am sure I shall be monstrous glad of Miss Marianne ' s company
context: is_fond
"is exceedingly fond"
han you did ; not but what she is exceedingly fond of YOU , but so it happened
"is monstrous fond"
so scornfully ! for they say he is monstrous fond of her , as well he may . I s
common_contexts shared by "monstrous" and "heartily": ['am_glad', 'be_glad']
context: am_glad
"am heartily glad"
ney enough to attempt it ." " I am heartily glad of it ," he cried . " May she
"am monstrous glad"
your sister is to marry him . I am monstrous glad of it , for then I shall have
usual noisy cheerfulness , " I am monstrous glad to see you -- sorry I could n
t however , as it turns out , I am monstrous glad there was never any thing in
possible that she should ." " I am monstrous glad of it . Good gracious ! I hav
context: be_glad
"be heartily glad"
saying now and then , ' I shall be heartily glad to hear she is well married .
"be monstrous glad"
Jennings , " I am sure I shall be monstrous glad of Miss Marianne ' s company
common_contexts shared by "monstrous" and "a": ['was_happy']
context: was_happy
"was a happy"
ne ' s behaviour as they travelled was a happy specimen of what future complais
"was monstrous happy"
thing of the kind . So then he was monstrous happy , and talked on some time ab
common_contexts shared by "monstrous" and "as": ['is_pretty']
context: is_pretty
"is as pretty"
, my dear , for shewing it me . It is as pretty a letter as ever I saw , and d
wing it me . It is as pretty a letter as ever I saw , and does Lucy ' s head a
"is monstrous pretty"
k how you will like them . Lucy is monstrous pretty , and so good humoured and
common_contexts shared by "monstrous" and "good": ['a_deal']
context: a_deal
"a good deal"
l ; but as she had already imbibed a good deal of Marianne ' s romance , witho
He and I have been at times thrown a good deal together , while you have been
th Mrs . Dashwood , so we shall be a good deal in Harley Street , I dare say ,
alone ," he replied , " for I have a good deal to say to you . This living of
"a monstrous deal"
e very genteel people . He makes a monstrous deal of money , and they keep thei
common_contexts shared by "monstrous" and "great": ['a_deal']
context: a_deal
"a great deal"
wn way , many cunning tricks , and a great deal of noise , as to outweigh all t
what you are doing . I have known a great deal of the trouble of annuities ; f
me as what belongs to this house . A great deal too handsome , in my opinion ,
inciple by my mother . I have seen a great deal of him , have studied his senti
, fat , elderly woman , who talked a great deal , seemed very happy , and rathe
een thirty and forty . He has seen a great deal of the world ; has been abroad
in all she does -- sometimes talks a great deal and always with animation -- bu
ey recommended the army . That was a great deal too smart for me . The law was
r , I dare say we should have seen a great deal of him in Somersetshire , if it
well married ! I hope you will be a great deal at Combe Magna . It is a sweet
old them it is all very true , and a great deal more . You will be delighted wi
ment at all ; indeed , she does me a great deal more harm than good , for I am
ings received the information with a great deal of joy , and many assurances of
ut it did not signify , for it was a great deal too far off to visit ; she hate
e in time . A thousand a - year is a great deal for a mother to give away , to
ied ; " but however there is still a great deal to be done . There is not a sto
not be forgotten .-- She must have a great deal to leave ." " Nothing at all ,
ious that everybody else should do a great deal ; and an offer from Colonel Bra
hinks just the same . He and I had a great deal of talk about it ; and the best
; so she told him directly ( with a great deal about sweet and love , you know
nd of whom therefore she must have a great deal to say , her mind was so much m
nfidant of himself , talked to her a great deal of the parsonage at Delaford ,
in the course of a twelve - month a great deal of instruction which I now feel
you , for as Colonel Brandon seems a great deal at home , nobody can tell what
"a monstrous deal"
e very genteel people . He makes a monstrous deal of money , and they keep thei
common_contexts shared by "monstrous" and "extremely": ['am_glad']
context: am_glad
"am extremely glad"
more than I can express ." " I am extremely glad to hear it , upon my word ; e
"am monstrous glad"
your sister is to marry him . I am monstrous glad of it , for then I shall have
usual noisy cheerfulness , " I am monstrous glad to see you -- sorry I could n
t however , as it turns out , I am monstrous glad there was never any thing in
possible that she should ." " I am monstrous glad of it . Good gracious ! I hav
common_contexts shared by "monstrous" and "remarkably": ['a_pretty']
context: a_pretty
"a remarkably pretty"
lexion , regular features , and a remarkably pretty figure . Marianne was stil
"a monstrous pretty"
. " Now , Palmer , you shall see a monstrous pretty girl ." He immediately went
common_contexts shared by "monstrous" and "sweet": ['a_pretty']
context: a_pretty
"a sweet pretty"
at his house ; but they say it is a sweet pretty place ." " As vile a spot as
"a monstrous pretty"
. " Now , Palmer , you shall see a monstrous pretty girl ." He immediately went
common_contexts shared by "monstrous" and "vast": ['a_deal']
context: a_deal
"a vast deal"
s convenience ; and it HAS cost me a vast deal of money ." " More than you thi
othing but only civility ?-- I saw a vast deal more . Such kindness as fell to
here come the Richardsons . I had a vast deal more to say to you , but I must
ell , I am convinced that there is a vast deal of inconsistency in almost ever
"a monstrous deal"
e very genteel people . He makes a monstrous deal of money , and they keep thei
common_contexts shared by "monstrous" and "amazingly": ['am_glad']
context: am_glad
"am amazingly glad"
l be in no hurry to be gone . I am amazingly glad you did not keep to YOUR WORD
"am monstrous glad"
your sister is to marry him . I am monstrous glad of it , for then I shall have
usual noisy cheerfulness , " I am monstrous glad to see you -- sorry I could n
t however , as it turns out , I am monstrous glad there was never any thing in
possible that she should ." " I am monstrous glad of it . Good gracious ! I hav
That output was generated by the following code.
from nltk.book import *
from contextlib import redirect_stdout
import io
def get_similar(text, word):
f = io.StringIO()
with redirect_stdout(f):
text.similar(word)
return f.getvalue().split()
def get_concordance(text, word):
f = io.StringIO()
with redirect_stdout(f):
text.concordance(word, lines=1000)
return f.getvalue().split(sep='\n')[1:-1]
def get_common_contexts(text, word_list):
f = io.StringIO()
with redirect_stdout(f):
text.common_contexts(word_list)
return f.getvalue().split()
def print_full_similars(text, word):
word_concordances = get_concordance(text, word)
similars = get_similar(text, word)
for similar in similars:
similar_concordances = get_concordance(text, similar)
common_contexts = get_common_contexts(text, [word, similar])
print('common_contexts shared by "' + word + '" and "' + similar + '":', common_contexts)
for context in common_contexts:
print('\tcontext:', context)
before, after = context.split('_')
print('\t\t"' + before, similar, after + '"')
for concordance in similar_concordances:
if (before + ' ' + similar + ' ' + after).lower() in concordance.lower():
print('\t\t\t'+ concordance)
print('\t\t"' + before, word, after + '"')
for concordance in word_concordances:
if (before + ' ' + word + ' ' + after).lower() in concordance.lower():
print('\t\t\t'+ concordance)
print()
print_full_similars(text2, 'monstrous')
Just change the last line to get other examples, as Moby Dick by Herman Melville 1851:
print_full_similars(text1, 'monstrous')
