jQuery uses pretty much the same principles as browsers themselves when they need to apply style rules from elements - scan all elements and check against all rules declared in CSS.
And so does jquery:
for each element in given subtree/input set do
if element matches selector do
add element to output set
end if
end for
return output set
So the task has computational complexity of O(N)
where N is a number of DOM elements in input set.
There are quite few tricks/optimizations possible but in general
it is as complex as that.
Some jQuery selectors have even worse complexity, for example this
$( "li:has(ul)")
selector has computation complexity of O(N*N)
so use with care.