It depends what you mean by "entire operating system": the most credible approach would be a hybrid OS which uses Java for the "user space" and something like Linux as the underlying OS for hardware access.
Some things to consider:
- Obviously the JVM itself needs to run on something. This might be a native JVM implementation running on another OS or it might be hardware, but either way it would need to use something other than pure Java at some point in the stack. Hence the suggestion of something like Linux to play this role.
- Unless you allow linking of extra native code, your interactions with the underlying hardware / platform would be dictated by what is available in the Java runtime environment (e.g. File abstractions etc. for file access). This is certainly enough to implement a working OS shell, but might not be enough to implement direct hardware access to many devices, for example.
- For a modern OS, you'd probably want to use OpenGL for the graphics (or a similar high performance graphics library). That's not in the standard Java runtime environment at present, so you would need to find a solution to include this (e.g. LWJGL)