Answering the title question (which originaly was: Could you say that nlgn equals O(n^2)? ):
No, you can't as nlgn
is a function and O(n^2)
is a set.
Answering your question from the body:
Well, yes, nlogn
is O(n^2)
... but don't try to answer each question on the exam with O(n^n)
. It's not what they're asking. Big-O notation isn't used for giving the best answer. It's just a way of giving some information.
According to Wikipedia:
Big O notation is a mathematical notation that describes the limiting behavior of a function when the argument tends towards a particular value or infinity. It is a member of a family of notations invented by Paul Bachmann, Edmund Landau, and others, collectively called Bachmann-Landau notation or asymptotic notation.