So, to first restate the question: Is there a problem that is in NP but neither in P, nor in NPC nor in NPH?
Let's start with the definition of the NP-complete class: A problem p
is NP-complete if it itself is NP and if every problem q
that is NP can be polynomially reduced to p
.
Next, the definition of the NP-hard class: A problem h
is NP-hard if there exists a problem p
that is NP-complete that can be polynomially reduced to h
.
What does the two definitions mean:
- NP-complete problems form a subset of NP which have the property that every NP problem can be polynomially reduced to any of them, including themselves.
- NP-hard problems don't need to be NP-complete. By polynomial reduction you can make problem "harder", but not "easier".
- From the definition of NP-hard, it follows that all NP problems are reducible to an NP-hard problem.
- Therefore, NP-hard problems are either NP-complete, or not NP at all.
Now back to the question - is there a problem that is NP but neither P, nor NP-complete, nor NP-hard?
First, the problem must be NP. That is easy from the definition of the NP class. It is the set of problems whose solutions can be verified polynomially. That means right away that such problem will not be NP-hard.
Second, the problem must not be P. That cannot be told as we don't know whether P = NP but let's assume that P != NP so we can therefore assume that there can, in fact, be a problem that is NP but not P.
Finally, we would need to show that the problem is not NP-complete. From the definition, NP-complete problems have the property that every NP problem can be reduced to it (more formally, that for every NP problem there is a polynomial reduction...). So the negation of that is that there is at least one NP problem that is not reducible (more formally, that there is no polynomial reduction...) to it. But showing that there is no reduction is, in my opinion, impossible.
So, finally, it is about our (your) ability to prove NP-completeness of such problem. You can prove that a problem is NP-complete if it is, but proving the opposite is, in my opinion, impossible.
I welcome all the more theoretically educated to edit my answer to be more accurate.