Stack Overflow
Stack Overflow

Questions tagged [text-segmentation]

Text segmentation is the process of dividing written text into meaningful units, such as words, sentences, or topics.

Text segmentation is the process of dividing written text into meaningful units, such as words, sentences, or topics.

References:

Related Tags:

197 questions
8
votes
7 answers

parsing words in a continuous string

If a have a string with words and no spaces, how should I parse those words given that I have a dictionary/list that contains those words? For example, if my string is "thisisastringwithwords" how could I use a dictionary to create an output "this…
locoboy
  • 38,002
  • 70
  • 184
  • 260
8
votes
1 answer

How do you customise text segmentation to not break between a digraph?

Works: #!/usr/bin/env python3 from uniseg.graphemecluster import grapheme_clusters def albanian_digraph_dh(s, breakables): for i, breakable in enumerate(breakables): if s.endswith('d', 0, i) and s.startswith('h', i): yield 0 …
daxim
  • 39,270
  • 4
  • 65
  • 132
8
votes
2 answers

Paragraph Segmentation using Machine Learning

I have a large repository of documents in PDF format. The documents come from different sources, and have no one single style. I use Tika to extract the text from the documents, and now I'd like to segment the text into paragraphs. I can't use…
Gino
  • 675
  • 2
  • 10
  • 20
8
votes
2 answers

Javascript implementation of UAX 29 Unicode Text Segmentation?

Is anyone aware of any JavaScript implementations of UAX #29, Unicode Text Segmentation? I'm specifically interested in Word Boundaries. I was hopeful when I came across XRegExp, but it seems to use the standard JavaScript implementation of \b.
Paul Butcher
  • 10,722
  • 3
  • 40
  • 44
8
votes
3 answers

How to split paragraphs into sentences?

Please have a look at the following. String[]sentenceHolder = titleAndBodyContainer.split("\n|\\.(?!\\d)|(?
PeakGen
  • 21,894
  • 86
  • 261
  • 463
7
votes
2 answers

Word splitting statistical approach

I want to solve word splitting problem (parse words from long string with no spaces). For examle we want extract words from somelongword to [some, long, word]. We can achieve this by some dynamic approach with dictionary, but another issue we…
mishadoff
  • 10,719
  • 2
  • 33
  • 55
7
votes
1 answer

difference between Tokenization and Segmentation

What is the difference between Tokenization and Segmentation in NLP. I searched about them but I didn't really find any differences .
Mahmoud Noor
  • 186
  • 12
7
votes
3 answers

How to get sentence number from input?

It seems hard to detect a sentence boundary in a text. Quotation marks like .!? may be used to delimite sentences but not so accurate as there may be ambiguous words and quotations such as U.S.A or Prof. or Dr. I am studying Tperlregex library and…
Warren
  • 795
  • 1
  • 10
  • 19
7
votes
2 answers

Java library that finds sentence boundaries

Does anyone know of a Java library that handles finding sentence boundaries? I'm thinking that it would be a smart StringTokenizer implementation that knows about all of the sentence terminators that languages can use. Here's my experience with…
Mike Sickler
  • 33,662
  • 21
  • 64
  • 90
7
votes
5 answers

Independent clause boundary disambiguation, and independent clause segmentation – any tools to do this?

I remember skimming the sentence segmentation section from the NLTK site a long time ago. I use a crude text replacement of “period” “space” with “period” “manual line break” to achieve sentence segmentation, such as with a Microsoft Word…
Jeff Kang
  • 279
  • 4
  • 13
7
votes
2 answers

How to iterate through sentence of string in Python?

Assume I have a string text = "A compiler translates code from a source language". I want to do two things: I need to iterate through each word and stem using the NLTK library. The function for stemming is PorterStemmer().stem_word(word). We have…
ChamingaD
  • 2,908
  • 8
  • 35
  • 58
6
votes
4 answers

Highlighting long sentences using jQuery

I'd like to highlight long sentences (say, 50 words or greater) contained in an array of paragraph objects on a page, ie $("#content p"). I'm not sure how to tackle this. I originally tried to highlight all sentences, but ran in trouble when they…
The Wilky Bar Kid
  • 164
  • 8
6
votes
2 answers

Word segmentation using dynamic programming

So first off I'm very new to Python so if I'm doing something awful I'm prefacing this post with a sorry. I've been assigned this problem: We want to devise a dynamic programming solution to the following problem: there is a string of characters…
xe0
  • 81
  • 1
  • 7
6
votes
5 answers

Formatting sentences in a string using C#

I have a string with multiple sentences. How do I Capitalize the first letter of first word in every sentence. Something like paragraph formatting in word. eg ."this is some code. the code is in C#. " The ouput must be "This is some code. The code…
AlwaysAProgrammer
  • 2,927
  • 2
  • 31
  • 40
5
votes
1 answer

Split text into sentences

I wish to split text into sentences. Can anyone help me? I also need to handle abbreviations. However my plan is to replace these at an earlier stage. Mr. -> Mister import re import unittest class Sentences: def __init__(self,text): …
Baz
  • 12,713
  • 38
  • 145
  • 268
1 2
3
…
13 14