Extract the k maximum elements of a list

Question

Let's say I have a collection of some type, e.g.

IEnumerable<double> values;

Now I need to extract the k highest values from that collection, for some parameter k. This is a very simple way to do this:

values.OrderByDescending(x => x).Take(k)

However, this (if I understand this correctly) first sorts the entire list, then picks the first k elements. But if the list is very large, and k is comparatively small (smaller than log n), this is not very efficient - the list is sorted in O(nlog n), but I figure selecting the k highest values from a list should be more like O(nk).

So, does anyone have any suggestion for a better, more efficient way to do this?

Answer 1

This gives a bit of a performance increase. Note that it's ascending rather than descending but you should be able to repurpose it (see comments):

static IEnumerable<double> TopNSorted(this IEnumerable<double> source, int n)
{
    List<double> top = new List<double>(n + 1);
    using (var e = source.GetEnumerator())
    {
        for (int i = 0; i < n; i++)
        {
            if (e.MoveNext())
                top.Add(e.Current);
            else
                throw new InvalidOperationException("Not enough elements");
        }
        top.Sort();
        while (e.MoveNext())
        {
            double c = e.Current;
            int index = top.BinarySearch(c);
            if (index < 0) index = ~index;
            if (index < n)                    // if (index != 0)
            {
                top.Insert(index, c);
                top.RemoveAt(n);              // top.RemoveAt(0)
            }
        }
    }
    return top;  // return ((IEnumerable<double>)top).Reverse();
}

Answer 2

Consider the below method:

static IEnumerable<double> GetTopValues(this IEnumerable<double> values, int count)
{
    var maxSet = new List<double>(Enumerable.Repeat(double.MinValue, count));
    var currentMin = double.MinValue;

    foreach (var t in values)
    {
        if (t <= currentMin) continue;
        maxSet.Remove(currentMin);
        maxSet.Add(t);
        currentMin = maxSet.Min();
    }

    return maxSet.OrderByDescending(i => i);
}

And the test program:

static void Main()
{
    const int SIZE = 1000000;
    const int K = 10;
    var random = new Random();

    var values = new double[SIZE];
    for (var i = 0; i < SIZE; i++)
        values[i] = random.NextDouble();

    // Test values
    values[SIZE/2] = 2.0;
    values[SIZE/4] = 3.0;
    values[SIZE/8] = 4.0;

    IEnumerable<double> result;

    var stopwatch = new Stopwatch();

    stopwatch.Start();
    result = values.OrderByDescending(x => x).Take(K).ToArray();
    stopwatch.Stop();
    Console.WriteLine(stopwatch.ElapsedMilliseconds);

    stopwatch.Restart();
    result = values.GetTopValues(K).ToArray();
    stopwatch.Stop();
    Console.WriteLine(stopwatch.ElapsedMilliseconds);
}

On my machine results are 1002 and 14.

Answer 3

Another way of doing this (haven't been around C# for years, so pseudo-code it is, sorry) would be:

highestList = []
lowestValueOfHigh = 0
   for every item in the list
        if(lowestValueOfHigh > item) {
             delete highestList[highestList.length - 1] from list
             do insert into list with binarysearch
             if(highestList[highestList.length - 1] > lowestValueOfHigh)
                     lowestValueOfHigh = highestList[highestList.length - 1]
   }

Answer 4

I wouldn't state anything about performance without profiling. In this answer I'll just try to implement O(n*k) take-one-enumeration-for-one-max-value approach. Personally I think that ordering approach is superior. Anyway:

public static IEnumerable<double> GetMaxElements(this IEnumerable<double> source)
    {
        var usedIndices = new HashSet<int>();
        while (true)
        {
            var enumerator = source.GetEnumerator();
            int index = 0;
            int maxIndex = 0;
            double? maxValue = null;
            while(enumerator.MoveNext())
            {
                if((!maxValue.HasValue||enumerator.Current>maxValue)&&!usedIndices.Contains(index))
                {
                    maxValue = enumerator.Current;
                    maxIndex = index;
                }
                index++;
            }
            usedIndices.Add(maxIndex);
            if (!maxValue.HasValue) break;
            yield return maxValue.Value;
        }
    }

Usage:

var biggestElements = values.GetMaxElements().Take(3);

Downsides:

1. Method assumes that source IEnumerable has an order
2. Method uses additional memory/operations to save used indices.

Advantage:

• You can be sure that it takes one enumeration to get next max value.

See it running

Answer 5

Here is a Linqy TopN operator for enumerable sequences, based on the PriorityQueue<TElement, TPriority> collection:

/// <summary>
/// Selects the top N elements from the source sequence. The selected elements
/// are returned in descending order.
/// </summary>
public static IEnumerable<T> TopN<T>(this IEnumerable<T> source, int n,
    IComparer<T> comparer = default)
{
    ArgumentNullException.ThrowIfNull(source);
    if (n < 1) throw new ArgumentOutOfRangeException(nameof(n));
    PriorityQueue<bool, T> top = new(comparer);
    foreach (var item in source)
    {
        if (top.Count < n)
            top.Enqueue(default, item);
        else
            top.EnqueueDequeue(default, item);
    }
    List<T> topList = new(top.Count);
    while (top.TryDequeue(out _, out var item)) topList.Add(item);
    for (int i = topList.Count - 1; i >= 0; i--) yield return topList[i];
}

Usage example:

IEnumerable<double> topValues = values.TopN(k);

The topValues sequence contains the k maximum values in the values, in descending order. In case there are duplicate values in the topValues, the order of the equal values is undefined (non-stable sort).

For a SortedSet<T>-based implementation that compiles on .NET versions earlier than .NET 6, you could look at the 5th revision of this answer.

An operator PartialSort with similar functionality exists in the MoreLinq package. It's not implemented optimally though (source code). It performs invariably a binary search for each item, instead of comparing it with the smallest item in the top list, resulting in many more comparisons than necessary.

Surprisingly the LINQ itself is well optimized for the OrderByDescending+Take combination, resulting in excellent performance. It's only slightly slower than the TopN operator above. This applies to all versions of the .NET Core and later (.NET 5 and .NET 6). It doesn't apply to the .NET Framework platform, where the complexity is O(n*log n) as expected.

A demo that compares 4 different approaches can be found here. It compares:

1. values.OrderByDescending(x => x).Take(k).
2. values.OrderByDescending(x => x).HideIdentity().Take(k), where HideIdentity is a trivial LINQ propagator that hides the identity of the underlying enumerable, and so it effectively disables the LINQ optimizations.
3. values.PartialSort(k, MoreLinq.OrderByDirection.Descending) (MoreLinq).
4. values.TopN(k)

Below is a typical output of the demo, running in Release mode on .NET 6:

.NET 6.0.0-rtm.21522.10
Extract the 100 maximum elements from 2,000,000 random values, and calculate the sum.

OrderByDescending+Take              Duration:   156 msec, Comparisons:  3,129,640, Sum: 99.997344
OrderByDescending+HideIdentity+Take Duration: 1,415 msec, Comparisons: 48,602,298, Sum: 99.997344
MoreLinq.PartialSort                Duration:   277 msec, Comparisons: 13,999,582, Sum: 99.997344
TopN                                Duration:    62 msec, Comparisons:  2,013,207, Sum: 99.997344