Ruby, Split Text Into Sentences

Question

Following a tutorial in a book, using the following code to split text into sentences,

def sentences
    gsub(/\n|\r/, ' ').split(/\.\s*/)
end

It works, but if theres a newline which began without a period preceding it, for example,

Hello. two line sentence
and heres the new line

theres a "\t" placed at the beginning of each sentence. So if i called the method on the above sentence i would get

["Hello." "two line sentence /tand heres the new line"]

Any help would be much appreciated! Thanks!

I think it's not clear what you are asking. What exactly you want to do and what is getting wrong? — Tamer Shlash, Aug 24 '14 at 19:43
So the method should split text into sentences based on a period followed by a whitespace. — user2085143, Aug 24 '14 at 19:44
So calling .sentences on the above lines should result in ["Hello", "two line sentence and heres the new line"] but i am getting a /t when theres a new line. — user2085143, Aug 24 '14 at 19:47
http://www.java2s.com/Code/Ruby/String/SplittingTextintoSentences.htm basically this. — user2085143, Aug 24 '14 at 19:49
I think that the origin of the problem is probably that the tab character was already here. You can avoid the problem using `gsub(/\s+/, ' ')` that is more radical. — Casimir et Hippolyte, Aug 24 '14 at 19:52

score 2 · Answer 1 · edited May 23 '17 at 12:33

Splitting text into sentences is best achieved using Stanford CoreNLP. In the example method provided in the question, any acronyms or name prefixes such as "Mr." would also be split.

The stanford-core-nlp ruby gem provides the ruby interface. See the instructions for installing the gem and Stanford CoreNLP in this answer, then you could write some code like this:

require "stanford-core-nlp"

StanfordCoreNLP.use :english
StanfordCoreNLP.model_files = {}
StanfordCoreNLP.default_jars = [
  'joda-time.jar',
  'xom.jar',
  'stanford-corenlp-3.5.0.jar',
  'stanford-corenlp-3.5.0-models.jar',
  'jollyday.jar',
  'bridge.jar'
]

pipeline =  StanfordCoreNLP.load(:tokenize, :ssplit)

text = 'Hello. two line sentence
and heres the new line'
text = StanfordCoreNLP::Annotation.new(text)
pipeline.annotate(text)
text.get(:sentences).each{|s| puts "sentence: " + s.to_s}

#output:
#sentence: Hello.
#sentence: two line sentence
#and heres the new line

Ruby, Split Text Into Sentences

1 Answers1