Fast lookup in a multidimensional System.Collection.IList

Question

I need your advice on the following.

I have a multi-dimensional IList containing items which have an index, Id and Text. Normally I know the value of Id and based on that I need to get the Text. Both Id and Text values are read from a database.

What we are currently using to get the value of Text field is:

foreach (Object myObj in List) 
{
    if (((MessageType)myObj).Id == id) 
    {
        return ((MessageType)myObj).Text;
    }
}

When count in IList becomes large (more than 32K), it takes some time to process.

Question: Is there a way to efficiently get the Text value without iterating through the IList?

Things I tried without success:

• Use List.IndexOf(Id) - did not work because IndexOf applies to text only.
• Converting List to multi-dimensional array - failed on List.CopyTo(array,0) my guess because it is multi-dimensional: string[] array=new string[List.Count,List.Count]; List.CopyTo(array,0);

I can not use a AJAX/JQuery solution because it is an existing(live) project and it will take too much to re-code.

Thanks

Answer 1

If you want fast searching by some identifier in a collection with 32k elements, you should use Dictionary<K,V> as your collection.

var dict = new Dictionary<IDType, MessageType>();

A Dictionary is basically a search tree where the elements are stored in a sorted way so an element with a specific key (in your case Id) can be found without looking at all elements. For more information see MSDN.

If you cannot refactor the collection to be a dictionary, you may initially fill the dictionary (slow) and then search in the dictionary (fast). This will only be faster if you do multiple searches before you fill the dictionary again, i.e. if your list does not change often.

foreach(object o in List)
{
    var msg = (MessageType)o;
    dict.Add(msg.Id, msg);
}

Searching then is easy:

MessageType msg = dict[id];

EDIT: Well, I was curious and wrote a test routine which compares the linear search and the dictionary approach. Here's what I used:

using System;
using System.Collections;
using System.Collections.Generic;
using System.Diagnostics;

namespace ConsoleApplication1
{
    class MessageType
    {
        public string Id;
        public string Text;
    }

    class Program
    {
        static void Main(string[] args)
        {
            var rand = new Random ();
            // filling a list with random text messages
            List<MessageType> list = new List<MessageType>();
            for (int i = 0; i < 32000; i++)
            { 
                string txt = rand.NextDouble().ToString();
                var msg = new MessageType() {Id = i.ToString(), Text = txt };
                list.Add(msg);
            }
            IList List = (IList)list;

            // doing some random searches
            foreach (int some in new int[] { 2, 10, 100, 1000 })
            {
                var watch1 = new Stopwatch();
                var watch2 = new Stopwatch();
                Dictionary<string, MessageType> dict = null;
                for (int i = 0; i < some; i++)
                {
                    string id = rand.Next(32000).ToString();
                    watch1.Start();
                    LinearLookup(List, id);
                    watch1.Stop();

                    watch2.Start();
                    // fill once
                    if (dict == null)
                    {
                        dict = new Dictionary<string, MessageType>();
                        foreach (object o in List)
                        {
                            var msg = (MessageType)o;
                            dict.Add(msg.Id, msg);
                        }
                    }
                    // lookup 
                    DictionaryLookup(dict, id);
                    watch2.Stop();
                }

                Console.WriteLine(some + " x LinearLookup took " 
                    + watch1.Elapsed.TotalSeconds + "s");
                Console.WriteLine("Dictionary fill and " + some 
                    + " x DictionaryLookup took " 
                    + watch2.Elapsed.TotalSeconds + "s");
            }
        }

        static string LinearLookup(IList List, string id)
        {
            foreach (object myObj in List)
            {
                if (((MessageType)myObj).Id == id)
                {
                    return ((MessageType)myObj).Text;
                }
            }
            throw new Exception();
        }

        static string DictionaryLookup(Dictionary<string, MessageType> dict,
            string id)
        {
            return dict[id].Text;
        }
    }
}

The results I got in Release / x86:

Number of | Time [ms] with | Time[ms] with | Speedup (approx.)
searches  | linear search  | dictionary(*) | with dictionary
----------+----------------+---------------+-----------------
      2   |      1.161     |   2.006       |   0.6
----------+----------------+---------------+-----------------
     10   |      2.834     |   2.060       |   1.4
----------+----------------+---------------+-----------------
    100   |     25.39      |   1.973       |   13
----------+----------------+---------------+-----------------
   1000   |    261.4       |   5.836       |   45
----------+----------------+---------------+-----------------

(*) including filling the dictionary once.

So, I was a bit optimistic to say that searching twice would already pay off. In my test application I have to search 10 times for the dictionary to be faster.

I'm sorry I could not make a more realistic example, my Ids are all sorted. Feel free to try modifying and experimenting though ;-)

Answer 2

From the looks of it you have a List<MessageType> here, which is not multi-dimensional. Rather the objects inside the list have multiple properties.

You could easily get them out with LINQ much faster than a loop most likely:

var text = (from MessageType msgType in myList
            where msgType.Id == id
            select msgType.Text).FirstOrDefault();

Or even easier with an inline LINQ statement:

var text = myList.Where(s => s.Id == id).Select(s => s.Text).FirstOrDefault();

NOTE: As mentioned in comments above, the speed of these LINQ statements are only as good as the object's position in the List. If it is the last object in the list, you will likely see the same performance discrepancy. Dictionary<Index, MessageType> is going to be much more performant.

Answer 3

Better way is to use ILookup. For example:

var look = query.ToLookup(x => x.SomeID, y=> y.Name)

and use:

if (look.Contains(myID)){
   var name = look[myID].First();
}