select top N for each category w/o sorting if there are less than N rows

Question

Given the following table, the question is to find for example the top N C2 from each C1.

C1 C2 
1  1
1  2
1  3
1  4
1  ...
2  1
2  2
2  3
2  4
2  ...
....

So if N = 3, the results are

C1 C2 
1  1
1  2
1  3
2  1
2  2
2  3
....

The proposed solutions use the window function and partition by

• Select top 10 records for each category

• https://www.the-art-of-web.com/sql/partition-over/

For example,

SELECT rs.Field1,rs.Field2 
FROM (
    SELECT Field1,Field2, Rank() 
      over (Partition BY Section
            ORDER BY RankCriteria DESC ) AS Rank
    FROM table
) rs WHERE Rank <= 3

I guess what it does is sorting then picking the top N.

However if some categories have less N elements, we can get the top N w/o sorting because the top N must include all elements in the category.

The above query uses Rank(). My question applies to other window functions like row_num() or dense_rank().

Is there a way to ignore the sorting at the case?

Also I am not sure if the underlying engine can optimize the case: whether the inner partition/order considers the outer where constraints before sorting.

Using partition+order+where is a way to get the top-N element from each category. It works perfect if each category has more than N element, but has additional sorting cost otherwise. My question is if there is another approach that works well at both cases. Ideally it does the following

for each category {
   if # of element <= N:
      continue
   sort and get the top N
}

For example, but is there a better SQL?

WITH table_with_count AS (
         SELECT Field1, Field2, RankCriteria, count() over (PARTITION BY Section) as c
         FROM table
),

rs AS (
    SELECT Field1,Field2, Rank() 
      over (Partition BY Section
            ORDER BY RankCriteria DESC ) AS Rank
    FROM table_with_count 
    where c > 10
) 

(SELECT Field1,Field2e FROM rs WHERE Rank <= 10)
     union
(SELECT Field1,Field2 FROM table_with_count WHERE c <= 10)

Answer 1

No, an there really shouldn't be. Overall what you describe here is the XY-problem.

You seem to:

• Worry about sorting, while in fact sorting (with optional secondary sort) is the most efficient way of shuffling / repartitioning data, as it doesn't lead to proliferation of file descriptors. In practice Spark strictly prefers sort over alternatives (hashing) for exactly that reason.
• Worry about "unnecessary" sorting of small groups, when in fact the problem is intrinsic inefficiency of window functions, which require full shuffle of all data, therefore exhibit the same behavior pattern as infamous groupByKey.

There are more efficient patterns (MLPairRDDFunctions.topByKey being the most prominent example) but these haven't been ported to Dataset API, and would require custom Aggregator It is also possible to approximate selection (for example through quantile approximation), but this increases the number of passes over data, and in many cases won't provide any performance gains.

Answer 2

This is too long for a comment.

There is no such optimization. Basically, all the data is sorted when using windowing clauses. I suppose that a database engine could actually use a hash algorithm for the partition by and a sort algorithm for the order by, but I don't think that is a common approach.

In any case, the operation is over the entire set, and it should be optimized for this purpose. Trying not to order a subset would add lots of overhead -- for instance, running the sort multiple times for each subset and counting the number of rows in each subset.

Also note that the comparison to "3" occurs (logically) after the window function. I don't think window functions are typically optimized for such post-filtering (although once again, it is a possible optimization).