10

This is an interview question: Find all (english word) substrings of a given string. (every = every, ever, very).

Obviously, we can loop over all substrings and check each one against an English dictionary, organized as a set. I believe the dictionary is small enough to fit the RAM. How to organize the dictionary ? As for as I remember, the original spell command loaded the words file in a bitmap, represented a set of words hash values. I would start from that.

Another solution is a trie built from the dictionary. Using the trie we can loop over all string characters and check the trie for each character. I guess the complexity of this solution would be the same in the worst case (O(n^2))

Does it make sense? Would you suggest other solutions?

Michael
  • 41,026
  • 70
  • 193
  • 341
  • Complexity of looping over all substrings checking hashes depends on your hash computation - there are Theta(n^2) substrings of average length not O(1), so you need to compute a partial hash that you can increment with one character at a time in order to keep O(n^2) overall. The same is true of the trie or DAWG lookup, of course, you'd want to descend gradually checking all strings starting from a given point, but it's probably more obvious that it's the right thing to do. – Steve Jessop Mar 02 '11 at 19:01
  • Walking the trie, starting from every possible start character and outputting all legal words as you find them seems plenty efficient; you stop looking as soon as you find a sequence of characters that can't possibly be a prefix of a word, and you can't do better than O(n^2) - it might be possible that every substring is valid, and there's O(n^2) of those. – Chris Nash Mar 02 '11 at 20:25

3 Answers3

6

The Aho-Corasick string matching algorithm which "constructs a finite state machine that resembles a trie with additional links between the various internal nodes."
But everything considered the "build a trie from the English dictionary and do a simultaneous search on it for all suffixes of the given string" should be pretty good for an interview.

Eugen Constantin Dinca
  • 8,994
  • 2
  • 34
  • 51
1

I'm not sure a Trie will work easily to match sub words that begin in the middle of the string.

Another solution with a similar concept is to use a state machine or regular expression. the regular expression is just word1|word2|.... I'm not sure if standard regular expression engines can handle an expression covering the whole English language, but it shouldn't be hard to build the equivalent state machine given the dictionary.

Once the regular expression is compiled \ the state machine is built the complexity of analyzing a specific string is O(n)

Ophir Yoktan
  • 8,149
  • 7
  • 58
  • 106
  • This is essentially the same as the trie solution. – biziclop Mar 02 '11 at 19:07
  • 1
    @biziclop- I've worked with a library containing a minimum-state DFA for all of English and it's substantially more compact than a standard trie. Yes, it's essentially the same as the trie, but it's much more memory-efficient. – templatetypedef Mar 03 '11 at 10:07
1

The first solution can be refined to have a different hash map for each word length (to reduce collisions) but other than that I can't think of anything significantly better.

biziclop
  • 48,926
  • 12
  • 77
  • 104