Would it make sense to use MemoryMappedFile to perform a search on large text files?

Question

I'm tasked with implementing a search function that will search through several large (couple MB) log files and return the lines that contain the keywords. Log files are constantly being added to the pool so the search has to be dynamic every time.

Would it make sense to create a MemoryMappedFile for each file and then iterate through each line, matching the keywords? If not, what would be a better way to go about it?

Any links to example code would be much appreciated.

There may be some helpful answers in this question: http://stackoverflow.com/questions/2095437/c-how-do-you-search-a-large-text-file-for-a-string-without-going-line-by-line — Jason Down, Jul 18 '11 at 14:06
You could also have a look at boyer-moore search algorithms to improve search speeds — Simon, Jul 18 '11 at 14:43

Roy Dictus · Answer 1 · 2011-07-18T14:55:37.503

1

Why not just create a properly structured index object tree in memory, optimized for searching?

EDIT: Added after some comments...

Could be something like this:

class Index
{
    public Dictionary<string, List<SourceFile>> FilesThatContainThisWord {get; set;}
    ...
}


class SourceFile
{
    public string Path {get; set;}
    ...
}


// Code to look up a term
var filesThatContainMonday = myIndex.FilesThatContainThisWord["Monday"];

edited Jul 18 '11 at 14:55

answered Jul 18 '11 at 14:07

Roy Dictus

32,551
8
60
76

This would be a lot of work, and may be overkill. The OP only has a few megabytes to search, so unless the search volume is very high, a simple in-memory string search is probably sufficient. – JSBձոգչ Jul 18 '11 at 14:18
That would then qualify as a properly structured index object tree, optimized for searching :-) – Roy Dictus Jul 18 '11 at 14:21
And so how would I go about doing that? – Jamie Curtis Jul 18 '11 at 14:37

score 1 · Accepted Answer · answered Jul 18 '11 at 14:36

Yes. A "couple of MB" is not very much, it easily fits in 2 GB.

You'll want to use the constructor that takes a mapping size because the file will grow in time. Also, I think you'll need to recreate the Accessor or Stream on each search, but I find MSDN a bit unclear here.

With a Stream, it's trivial to create a StreamReader, and read every line. The whole process is very likely I/O bound on reasonable hardware, so don't bother with CPU optimizations initially.

Would it make sense to use MemoryMappedFile to perform a search on large text files?

2 Answers2