how to perform query expansion

Question

I am working on a C# application where the user provides a set of words ( typically less than 10) and I need to retrieve all the synonyms of these words. This is my first time working with dictionary and these stuff. I need to know the steps to follow and if there an existing dictionary that provides synonyms that I can integrate with my application or if there is an open source application or code that I can use.

Do you have a dictionary to work with? Does it have to be a local dictionary, or is the use of an API allowed? — Pete Garafano, Mar 09 '13 at 17:30
googling "dictionary api" returns alot of results.. here one of them: http://www.dictionaryapi.com/ — Fredrik, Mar 09 '13 at 17:36
Dear TheGreatCO, I do not have already a dictionary to work with, and the dictionary should be local and I don't need exactly a dictionary, I need a sort of thesaurus to provide synonyms to a given word, some sort of query expansion. Dear FredrickRedin I did some research and all I did find is APIs and I don't need that. — user1905455, Mar 10 '13 at 22:16
@user1905455 when replying to a comment, be sure to @ respond to the user, so they get a notification. — Pete Garafano, Mar 10 '13 at 22:25
@user1905455 you need to take the first step of finding a thesaurus to download. I suspect you will need to find a dictionary and parse it to enable cross referencing. There are a few options, including princeton wordnet and project Gutenberg. — Pete Garafano, Mar 10 '13 at 22:36

Pete Garafano · Accepted Answer · 2013-03-12T13:58:59.113

To answer your first question. You can find a thesaurus download here: http://wordpresscloaker.com/blog/download-free-english-thesaurus-format-txt.html

I make no promises to the quality, accuracy, legality, licensing for use, or completeness of that file. However, this will get you on your way. You need to extract the mthesaur.txt and add it to your project folder.

Next, you need to read in the text file by doing the following:

var reader = new StreamReader(File.OpenRead(@"C:\mthesaur.txt"));
var dict = new Dictionary<string, string>();
while (!reader.EndOfStream)
{
    // Read the file line by line.
    var line = reader.ReadLine();

    // If the line isn't null, we can use it.  This shouldn't happen but it is a good sanity check.
    if (line == null) continue;
    // Split the line by the delimiter (a comma) so we can get the main word, the first one on the line.
    var splitLine = line.Split(',');
    var mainWord = splitLine[0];
    // To save us from having to loop through and only get the indexes above 0 (eg, skip the main word) we will just simply remove it from the line so we have just synonyms.
    line = line.Replace(mainWord + ",", string.Empty);
    // Now we make use of the dictionary type in C# and add the mainword as the key and the synonyms as the value.
    try
    {
        dict.Add(mainWord, line);
    }
    catch (ArgumentException argEx)
    {
        Console.WriteLine("Attempted to add {0} to the dictionary but it already exists.", mainWord);
    }
}

Now that we have everything in a key/value dictionary in C#, you can use LINQ to query out the synonyms for an entered word. This can be done by either using a drop down that contains all the key values from the dictionary (not recommended as this will be an extremely large drop down and hard to navigate for the user), a ListBox (better, easier to navigate), or a plain text search box. While this doesn't completely answer your question as there is nothing here about handling a GUI for the user, this should get you well on your way.

I have tried your code but when I compile it I get an ArgumentException on the last line and it says "An item with the same key has already been added." any idea on what went wrong? — user1905455, Mar 12 '13 at 09:28
@user1905455 I have updated the answer to catch this exception without crashing the program. However I suggest you put a break point on the `Console.WriteLine` to see which keys in particular cause the issue. Then I suggest you check the actual thesaurus file to see if this is a programming error or a problem with the code. — Pete Garafano, Mar 12 '13 at 14:00
thank you this code worked great and I found the repeated key — user1905455, Mar 12 '13 at 18:01

score 0 · Answer 2 · answered Mar 10 '13 at 23:07

If you use SQL full text search or the underlying technology - Microsoft Search Server (there is a free Express SKU) you will find thesaurus for multiple languages and other natural language processing tools. I am of course assuming you are working on an actual project, not on homework...

If you are more into open source, check out Lucene.net - it provides a search engine and I'm pretty sure it has thesaur

how to perform query expansion

2 Answers2