1

I know about the system calls that OS provides to protect programs from accessing other programs memory. But that can only help if I have used the system call library provided by OS. What if I write a assembly code myself that sets CPU bit for kernel mode and executes a privileged instruction ( let's say modify OS' program segment in memory ). Can OS protect against that ? P.S. Out of curiosity question. If any good blog or book reference can be provided, that would be helpful as I want to study OS in as much detail as possible.

Rishi Sharma
  • 35
  • 3
  • 1
    The OS puts a standard process in a mode where such instructions will cause exceptions. The memory is also protected, including disabling the ability to run code from data areas. However, debugger type processes can override all of this. – rcgldr Dec 10 '16 at 23:18
  • How does OS set that mode where such instructions will cause exceptions ? Once OS has handed over the CPU to a process then the process is allowed to do anything. So can't the process set the CPU flag for privileged mode and execute any instruction – Rishi Sharma Dec 10 '16 at 23:33
  • It depends on the processor, but generally programs are run in a restricted mode that prevents them from changing out of restricted mode. – rcgldr Dec 11 '16 at 00:43

2 Answers2

1

The processor protects again such malicious mischief by (1) requiring you to be in an elevated mode (for our example here, KERNEL); and (2) limiting access to kernel mode.

In order to enter kernel mode from user mode there either has to be an interrupt (not applicable here) or an exception. Usually both are handled the same way but there are some bizarre processors (Did anyone say Intel?) that do things a bit differently

The operating system exception and interrupt handlers must limits what the user mode program can do.

What if I write a assembly code myself that sets CPU bit for kernel mode and executes a privileged instruction

You cant just set the kernel mode bit in the processor status register to enter kernel mode.

Can OS protect against that ?

The CPU protects against that.

If any good blog or book reference can be provided, that would be helpful as I want to study OS in as much detail as possible.

The VAX/VMS Systems Internals book is old but it is cheap and shows how a real OS has been implemented.

user3344003
  • 20,574
  • 3
  • 26
  • 62
  • So if CPU protects against a program setting kernel mode bit, how does it let only OS set the kernel mode ? how does it differentiates whether an OS wants to set the kernel mode or a standard program.l ? – Rishi Sharma Dec 11 '16 at 05:38
  • @RishiSharma: OS runs on system first; It setup all the things in higher privilege and then ask the processor(core) to execute programs or user programs in user ring (less privileged). while OS keeps in running what we call protected mode. – amaneureka Dec 11 '16 at 14:53
0

This blog clearly explains what my confusion was. http://minnie.tuhs.org/CompArch/Lectures/week05.html Even though user programs can switch to kernel mode, but they have to do it through a interrupt instruction ( int in case x86) and for this interrupt, the interrupt handler is written by the OS. ( probably when it was in kernel mode at bootup time). So this way all priviliged instructions can only be executed by the OS code only.

Rishi Sharma
  • 35
  • 3