Python: find common sentence fragments in list of sentences

Question

Say I have a list of sentences like:

quick brown blah red work word
quick brown too red blah someone
quick gray one two three
quick gray two three four
quick gray johnson week summer
quick gray johnson day week water fall
quick gray wicked stopper fall
quick gray hotel flamer walk
doggie bone
doggie python
doggie python tree flower python
doggie python flower whatever
tree bone stick

I'm looking for code to return a list of the common 'parent' sentences:

quick brown
quick gray
quick gray johnson
doggie bone
doggie python
tree bone stick

THX

I have not tried anything - I figure I need some sort of recursive function. Before I break my head, are there any Python libraries out there...? — zippy242, Sep 13 '18 at 10:58

score 0 · Answer 1 · answered Sep 13 '18 at 10:36

Here you go:

def removeNumbers(data):
    result = []
    for sent in data:
        temp = []
        words = sent.split()
        for word in words:
            try:
                number = int(word)
                break
            except:
                temp.append(word)
        result.append(" ".join(temp))
    return result
data = [
    'quick brown 580 650 040 050',
    'quick brown 650 160 150 500',
    'quick gray 075 060 400',
    'quick gray 087 565 600',
    'quick gray johnson 149 135',
    'quick gray johnson 600 650 070 600',
    'quick gray 565 070 250',
    'quick gray 630 550 400',
    'doggie 256',
    'doggie python',
    'doggie python 350 675 106',
    'doggie python 417 560',
    'tree 196 106'
]
data = removeNumbers(data)
print(list(set(data)))

score 0 · Answer 2 · answered Sep 13 '18 at 10:46

You can easily do it with regex:

>>> result=[]
>>> for i in data:
>>>     r = re.search(r'([a-z]+\s*)+', i)
>>>     if r:
>>>         res = r.group(0).strip()
>>>         if res not in result:
>>>             result.append(res.strip())
>>> print(result)
['quick brown', 'quick gray', 'quick gray johnson', 'doggie', 'doggie python', 'tree']

Python: find common sentence fragments in list of sentences

2 Answers2