Search String By SubWords

Question

What Kind of algorithms + data structures that would help me to do that?

Having a file contains like 10000~ lines loaded in memory in a ordered set. With a given search string I want to be able to get all the lines that have words prefixed with words found in search string. Well let me give an example to clarify this:

Lines:

1. "A brow Fox flies."
2. "Boxes are full of food."
3. "Cats runs slow"
4. "Dogs hates eagles"
5. "Dolphins have eyes and teath"

Cases 1:

search string = "fl b a"

"A brow Fox flies."

• Explanation: search string have three words "fl", "b", and "a" and the only string that have some words that are prefixed with words from the search string is line 1.

Case 2:

search string "e do ha"

"Dogs hates eagles", "Dolphins have eyes and teath"

Solution

(fast enough for me it took about 30ms~(including sorting the final result) on my pc on a set of 10k lines 3 words each line)

• I used a trie suggested in answer.
• And some other hacky methods to be able to filter out duplicate and false positive results (mainly used hash sets for this).

Answer 1

I think what you're probably wanting is a trie. Construct one for the set of all words in your document, and have each leaf point to a hashset containing the indices of the lines in which the key of the leaf appears.

To execute a search, you'd use each fragment of the search string to navigate to a node in the tree and take the union over the hashsets of all leaves in that node's subtree. Then you'd take the intersection of those unions over the set of fragments to get the list of lines satisfying the search string.

Answer 2

Here is my 2 cents:

class DicVal
{
    public int OriginalValue;
    public int CurrentValue;

    public int LineNumber;
}

private static void Main()
{
    var a = "A brow Fox flies.\r\n" +
            "Boxes are full of food.\r\n" +
            "Cats runs slow\r\n" +
            "Dogs hates eagles\r\n" +
            "A brow Fox flies. AA AB AC\r\n" +
            "Dolphins have eyes and teath";

    var lines = a.Split(new[] {Environment.NewLine}, StringSplitOptions.RemoveEmptyEntries);
    var dic = new Dictionary<string, DicVal>
        {
            {"fl", new DicVal { OriginalValue = 1, LineNumber = -1}},
            {"b", new DicVal { OriginalValue = 1, LineNumber = -1}},
            {"a", new DicVal { OriginalValue = 4, LineNumber = -1}}
        };

    var globalCount = dic.Sum(x => x.Value.OriginalValue);
    var lineNumber = 0;

    foreach(var line in lines)
    {
        var words = line.Split(' ');
        var currentCount = globalCount;

        foreach (var word in words.Select(x => x.ToLower()))
        {
           for (var i = 1; i <= word.Length; i++)
           {
               var substr = word.Substring(0, i);
               if (dic.ContainsKey(substr))
               {
                   if (dic[substr].LineNumber != lineNumber)
                   {
                       dic[substr].CurrentValue = dic[substr].OriginalValue;
                       dic[substr].LineNumber = lineNumber;
                   }

                   if (dic[substr].CurrentValue > 0)
                   {
                       currentCount--;
                       dic[substr].CurrentValue--;
                   }
               }
           }
        }

        if(currentCount == 0)
            Console.WriteLine(line);

        lineNumber++;
    }
}

Not going to explain much, as code is the best documentation :P.

Output: A brow Fox flies. AA AB AC

Assuming you implement everything efficiently, the running time will be as good as possible, since you need to read every word at least ONCE.

Further optimization can be done and apply threading. You can view into PARALLEL AGGREGATION concept, as this problem can be parallelized easily.

Answer 3

Here's a fairly simple implementation that should be appropriate for your use case. The idea is that you can store all combinations of short prefixes for each line (and for each query) since you only have 10,000 lines and assuming each line doesn't contain too many words. Now look up each hash generated for the query string. For each hash match, we then check for an exact match. For my example code I consider only prefixes of length 1, however you could repeat this approach for prefixes of length 2 & 3 provided the prefixes in your query have those lengths too.

__author__ = 'www.google.com/+robertking'

from itertools import combinations
from collections import defaultdict

lines = [
    "A brow Fox flies.",
    "Boxes are full of food.",
    "Cats runs slow",
    "Dogs hates eagles",
    "Dolphins have eyes and teath"
 ]
lines = [line.lower() for line in lines]

def short_prefixes(line):
    for word in line.split():
        yield word[:1]

def get_hashes(line):
    starts = list(short_prefixes(line))
    for prefixes_in_hash in range(1, min(4, len(starts))):
        for hash_group in combinations(starts, r=prefixes_in_hash):
            yield tuple(sorted(hash_group))


def get_hash_map():
    possible_matches = defaultdict(list)
    for line_pos, line in enumerate(lines):
        for hash in get_hashes(line):
            possible_matches[hash].append(line_pos)
    return possible_matches


possible_matches = get_hash_map()

def ok(line, q):
    return all(line.startswith(prefix) or ((" " + prefix) in line) for prefix in q)

def query(search_string):
    search_string = search_string.lower()
    q = search_string.split()
    hashes = set(get_hashes(search_string))
    possible_lines = set()
    for hash in hashes:
        for line_pos in possible_matches[hash]:
            possible_lines.add(line_pos)

    for line_pos in possible_lines:
        if ok(lines[line_pos], q):
            yield lines[line_pos]

print(list(query("fl b a")))
#['a brow fox flies.']