Big-O notation is about the limiting behavior of the function as n increases without bound. There are a lot of "shortcuts" you can take in finding the asymptotic complexity of a function that are not valid algebraic steps in that they do not result in a function with the same value (but merely the same asymptotic behavior).
For example, in (n^2 log n + 10)/(n - 10), imagine picking a very large value for n, like 2^100. If you pick a large enough value, n-10 will be virtually indistinguishable (in terms of sheer size) from n, right? And n^2 log n will definitely be big enough to dwarf that puny +10. So as n increases, the value of that function gets asymptotically closer to the value of (n^2 log n)/n which is just n log n.
So your belief
there is no way to remove the 10s
is false.
meowgoesthedog's answer shows how you can prove that f(n) is O(n log n) a little more formally. It's worth noting that the argument given in the paragraph above is not mathematically rigorous. Often, when you're trying to prove something, you have to make an intuitive leap to the answer, and then show that the assumption must be true with more formal means.