You seem to be confused about how virtual memory and overcommitting are related.
Physical memory consists of the RAM chips in your computer. It is physically impossible to ever use more memory at the same time than you have physical memory. However, since the late 1970s out thereabouts there has been virtual memory, wherein the system will move some data from physical memory to disk, thereby storing memory that isn't actually being used away so physical memory can be used for other things. When this happens, programs cannot immediately use the data that was swapped out; when they try to do so anyway, the processor will generate a page fault, causing the operating system to load the required data from swap back into physical memory.
By using virtual memory, the total amount of in-flight data that can be in use is extended to the size of the physical memory plus the swap space. While this allows the system to run more programs at the same time, the amount of memory actually in use can never exceed the total available virtual memory (i.e., RAM + swap space)
In principle, the kernel should keep track of the amount of memory that programs have requested from it, and it should reject requests for more memory once its bookkeeping tells it that all memory has been allocated, even though not all of it may have been used. Most other kernels do this, and so they don't have an OOM killer (because they don't need one).
However, the system call for a program to allocate memory from the kernel doesn't allocate memory by the byte; it allocates it by larger blocks. In contrast, the actual implementation used by most programs to get memory (the malloc() C library function) does allow programs to allocate memory by the byte. What this means is that most programs, when they ask for some memory, will end up with malloc() allocating more than what they actually need. In addition, a good malloc() implementation will use some heuristics to keep smaller and larger allocations separate, so that memory fragmentation is not to bad, but doing so requires it to request more memory from the kernel, exacerbating the problem. As a result of these and other effects, without overcommitting, there will be large amounts of memory which are allocated but will never be used.
The idea of overcommitting is that you can safely allocate some more memory than the total amount of virtual memory your system has available without it backfiring. In essence, the kernel claims it allows an allocation, even though it knows it cannot live up to that promise, in the assumption that it won't need to live up to the full extent of those promises. However, the exact amount that can be overcommitted -- allocated -- without things ever going wrong cannot be predicted; so when the kernel detects that programs are trying to use all the memory they were allocated, the kernel will have to renege on its promises. This is where the OOM killer comes in.
Whether that is a good idea is up for debate. When the system runs out of memory, something is going to go wrong. If that's a case of some application trying to allocate more memory but the kernel having reserved everything already, then possibly that particular application is going to crash. If that just happens to be your X server, you're shit outta luck. At least if there's an OOM killer, it can decide which process to kill rather than things going bad after a bad allocation; and the OOM killer has some logic to avoid killing important processes, like your X server...
However, it you prefer, there are some sysctl knobs you can use to turn off overcommitting, it you think it's a bad idea.
