search string for longest keyword

Question

I'm using java, and have a large-ish (~15000) set of keywords (strings), and I have a document (string) that contains these keywords periodically.

I'd like to find the indices of each use of the keywords in the document, with a preference to longer keywords (ones with the most characters). For example, if my keywords were "water", "bottle", "drank", and "water bottle", and my document were "I drank from my water bottle", I'd like a result of:

2 drank

16 water bottle

My initial attempts were to use a trie, and go through the document character-by-character, and whenever a substring matches a keyword, record the initial index. However some of the keywords are prefixes for longer keywords (for example, "water" and "water bottle"), and the code would never find the longer one, as it would record "water"'s index, and then start over.

If it matters, the keywords may contain lower case letters, upper case letters, spaces, hyphens, and apostrophes (and capitalization matters).

So, any help in finding the longest keywords would be much appreciated. Thanks.

Answer 1

If keywords can be built up from smaller keywords, then all you have to do with your code that works is check the longer keywords first. Just a note: I didn't test this at all, I think I put enough work into this problem already! If this helps you don't forget to upvote + accept.

i.e.

import java.util.TreeSet;
import java.util.Comparator;
import java.util.LinkedList;
import java.util.HashMap;
import java.util.Iterator;

public class KeywordSearcher {
    private TreeSet<String> ts;

    public KeywordSearcher() {
    ts = new TreeSet<String>(new Comparator<String>() {
    // Sort all the keywords by length, largest first
        public int compare(String arg0, String arg1) {
            if(arg0.length() > arg1.length()) return -1;
            if(arg0.length() == arg1.length()) return 0;
            return 1;
        }});
    }

    public void addKeyword(String s) {
        ts.add(s);
    }

    private LinkedList<Integer> findKeyword(String document, String s) {
        int start = 0;
        int index;
        LinkedList<Integer> indexes = new LinkedList<Integer>();        

        while(true) {
            index = document.indexOf(s, start);
            if (index == -1) break;
            indexes.add(index);
            start = index + s.length();
        }

        return indexes;
    }

    public HashMap<String, LinkedList<Integer>> findAllKeywords(String document) {
        Iterator<String> is = ts.iterator();
        HashMap<String, LinkedList<Integer>> allIndices = new HashMap<String, LinkedList<Integer>>();

        while(is.hasNext()) {
            String nextKeyword = is.next();
        // See if we found a larger keyword, if we did already, skip this keyword
        boolean foundIt = false;
        for (String key : allIndices.keySet()) {
                if(key.contains(nextKeyword)) {
                    foundIt = true;
                    break;
                }
        }
            if (foundIt) continue;

            // We didn't find the larger keyword, look for the smaller keyword
            LinkedList<Integer> indexes = findKeyword(document, nextKeyword);

            if (indexes.size() > 0) allIndices.put(nextKeyword, indexes);
        }

        return allIndices;
    }
}

Answer 2

If I understand you correctly, you want to skip searching for "water" if you find "water bottle" in the document. That implies some sort of tree structure for your keywords.

My suggestion would be to arrange your keywords on a sorted tree like this:

drank
water bottle
    bottle
    water

In your code, you would search first for the terms that are at the root ("drank" and "water bottle"). If the number of matches for "water bottle" comes up zero, then you would navigate to the next level and search those terms ("bottle" and "water").

Creating the tree would require a bit of work.

But with this tree structure, you can have multiple compound words.

clean water bottle
    clean bottle
        clean
    water bottle
        bottle
        water