To put suggestion in one place.
Idea is to use divide and conquer strategy. If we can find pareto point P which splits rest of pareto points in half's in O(n) than it can be done in O(n log(h)). That can be checked by splitting points in three parts right of P, left and up of P, left and down of P. Third set doesn't have pareto points. And recursively do same procedure on other two set of points. Set that three parts split points in ratio: a, b, 1 - a - b
. With that complexity is:
T(n, h) = T(a*n, h/2) + T(b*n, h/2) + O(n)
<= T(n, h/2) + O(n)
<= T(n, h/4) + 2 * O(n)
<= T(n, h/8) + 3 * O(n)
<= ...
<= O(n log(h))
Problem is finding pareto point with that characteristic in <= O(n). Some simple heuristic like P where x >= (min(x) + max(x)) / 2, probably makes very good average.
I am not sure, but I think it is possible to use k-th order statistic to find point with that characteristic. Similar to finding median as pivot in quicksort.