0

I need to filter out several hundred "stop" words from a string. As there are many "stop" words I don't think it would be a good idea to do something like this:

sentence.replace(/\b(?:the|it is|we all|an?|by|to|you|[mh]e|she|they|we...)\b/ig, '');

How can i create something like a hash map to store stop words? In this map a key will be a stop word itself and value is not important. Then filtering will result in checking if the word is not present in stop words map. What data structure to use to build such a map?

ChaosPandion
  • 77,506
  • 18
  • 119
  • 157
dokondr
  • 3,389
  • 12
  • 38
  • 62
  • Relevant: [John Resig's Series on Dictionary Lookups](http://ejohn.org/blog/revised-javascript-dictionary-search/) – John Strickler Feb 22 '12 at 22:01
  • How many of these "stop words" do you have? The answer could be important. – ChaosPandion Feb 22 '12 at 22:03
  • possible duplicate of [JavaScript code to filter out common words in a string](http://stackoverflow.com/questions/6686718/javascript-code-to-filter-out-common-words-in-a-string). Note the solution builds a dictionary from a string. If you start with a dictionary, you can skip that part. – outis Feb 22 '12 at 22:06

1 Answers1

1

There's nothing that can beat regular expressions for this kind of job. However, there are two problems with them - hard to maintain (what you pointed to in your post) and performance problems with very large ones. I don't know how many alternatives one single regexp can handle, but I guess up to 20-30 would be fine in any case.

So you need some code that builds regular expressions dynamically from some data structure, which can be an array or just a string. I personally would prefer the sting, because it's easiest to maintain.

// taken from http://www.ranks.nl/resources/stopwords.html
stops = ""
+"a about above after again against all am an and any are aren't as  "
+"at be because been before being below between both but by can't    "
+"cannot could couldn't did didn't do does doesn't doing don't down  "
+"during each few for from further had hadn't has hasn't have        "
+"haven't having he he'd he'll he's her here here's hers herself     "
+"him himself his how how's i i'd i'll i'm i've if in into is isn't  "
+"it it's its itself let's me more most mustn't my myself no nor     "
+"not of off on once only or other ought our ours ourselves out      "
+"over own same shan't she she'd she'll she's should shouldn't so    "
+"some such than that that's the their theirs them themselves then   "
+"there there's these they they'd they'll they're they've this       "
+"those through to too under until up very was wasn't we we'd we'll  "
+"we're we've were weren't what what's when when's where where's     "
+"which while who who's whom why why's with won't would wouldn't     "
+"you you'd you'll you're you've your yours yourself yourselves      "

// how many to replace at a time
reSize = 20 

// build regexps
regexes = []
stops = stops.match(/\S+/g).sort(function(a, b) { return b.length - a.length })
for (var n = 0; n < stops.length; n += reSize)
    regexes.push(new RegExp("\\b(" + stops.slice(n, n + reSize).join("|") + ")\\b", "gi"));

Once you've got this, the rest is obvious:

regexes.forEach(function(r) {
    text = text.replace(r, '')
})

You need to experiment with reSize value to find out the optimal balance between the regex length and the total number of regexes. If the performace is critical, you can also run the generation part once and then cache the results (i.e. generated regexps) somewhere.

georg
  • 211,518
  • 52
  • 313
  • 390
  • Looking up words in a hash map will be slower then using regex? Can a plain JS object be considered as a hash map, where property names are map keys and property values are map values? – dokondr Feb 23 '12 at 09:46
  • Depends on what exactly you're doing, but yes, in most cases replacing 20 words at once will be faster than locating/eliminating them one by one. Yes, javascript objects are hash maps. – georg Feb 23 '12 at 10:57
  • Ok, yet how regex replacement of 20 words may be implemented in JS run-time? IMO there will be two arrays: 'sentence' and 'stopWords'. Every word in sentence will be compared with every word in stopWords anyway. Does not look awfully efficient to me. – dokondr Feb 23 '12 at 11:15