Memory access violations are a large topic :)
The Protection of Information in Computer Systems (from 1973 :) lays out of a mechanism of segments, where processes are allocated a base and a bound; any attempt to access memory outside the range base:base+bound
meant the program had done something silly and should be killed.
The 80x86 line of processors implement basic segment support, and the GEMSOS security kernel is an A1-certified operating system kernel based on this mechanism.
But segments aren't very dynamic, and almost all modern operating systems are paging systems, that page in memory when it isn't available. This relies on the CPU having an MMU, memory management unit, that checks all memory accesses for correct privileges and presence/absence of the correct memory mapping. When a process tries to access memory that isn't currently mapped into RAM, the MMU signals the CPU that a fault has occurred, and the CPU suspends the process to load the requested memory page from disk. (Or, if the memory should not be mapped for the process, say it tries to access 0x0
or some random memory location that hasn't been mapped with mmap
or similar memory allocating primitives, it kills the process.)
Intel's 80386 was the first Intel chip to support paging, which is why Windows 3.1's "386 Enchanced Mode" was so much better than the 286 mode.
Compilers aren't really involved, but the CPU, MMU, and operating system kernel must all work together.