-3

Is there a way to drop stop words (e.g., 'of' 'a' 'the' etc.) before using a JAVA based document classifiers (such as OpenNLP) etc. Or if you are doing it yourself (with JAVA) what could be the most efficient way to do it (Given that string comparison is inefficient). Also, Given that each document itself is not that big, i.e.,on average around 100 words, but the number of documents is assumed to be large. 

E.g., 
// Populate the stop words to a list
List<String> stopWordsList = ArrayList<>();

// Iterate through a list of documents
String currentDoc = getCurrentDoc();

String[] wordsArray = currentDoc.split(" ");    

 for ( String word : wordsArray ) {

      if (stopWordsList.contains(word)){
           // Drop it
      }
  }
dhg
  • 52,383
  • 8
  • 123
  • 144
Lahiru Sandakith Gallege
  • 1
  • 4

3 Answers3

0

Your technique is fine. However, you should make your stopWordsList a Set, not a List, so that you can look things up in constant time instead of linear time. In other words, you don't want to have to look through the whole stopWordsList to see if word is in there; you want to just see if it's in the set right away.

dhg
  • 52,383
  • 8
  • 123
  • 144
-1

You can try the following code : 

    String sentence = "This is a sample sentence for testing stop word deletion";

    String pattern = " a | the | for | is ";
    sentence = sentence.replaceAll(pattern, " ");

Result : This sample sentence testing stop word deletion

The pattern contains all the stop words separated by pipeline, to say that the pattern may contain either of those. Remember to have the spaces around the stop words to distinguish them as exact words. If not for the spaces it will replace all occurrences of the stop word's character combination even within words.

maheeka
  • 1,983
  • 1
  • 17
  • 25
  • Stop words can be upper/lower case, + your pattern doesn't work if the stop word is not wrapped between two white spaces, which could be the case if the sentence starts or ends with a stopword, or if followed by a comma, etc... – cheseaux Oct 15 '14 at 09:13
  • Noted. But most of the above mentioned could be solved with proper regex pattern, or a series of regex patterns for that matter. – maheeka Oct 15 '14 at 23:55
-2

No need to split, simply replace the target string with an empty string

String currentDoc = getCurrentDoc();
currentDoc = currentDoc.replace(stringToReplace,"");

Or, go with regex using replaceAll if you have multiple words to replace.

C.B.
  • 8,096
  • 5
  • 20
  • 34
  • Um, no, this is not a good idea. "a" is a stopword. Simply doing a replace would turn "apple" into "pple", which is clearly not what you want. – dhg Jul 17 '14 at 03:37