Quick way to have a minimal X install with minimal user interface/window manager and vnc:
apt-get install xserver-xorg-video-dummy vnc4server x11-xserver-utils xterm wm2
Start vnc, on the server in the account you want to use run:
vnc4server
enter password
** begin optional **
Optionally you may want to quit vnc and edit it's configuration and run it again. Use whatever options you want, this example is about as minimalistic as it can get (well, you could just run a terminal emulator without a window manager and use "fancy" X commands to manage windows yourself... ;-)
On the server:
killall vnc4server <-- or find the PIDs yourself and kill
vim $HOME/.vnc/xstartup
Add:
x-terminal-emulator -geometry 80x24+10+10 -ls -title "$VNCDESKTOP Desktop" & wm2 &
Run vnc4server again.
** end optional ***
On the computer which you want to use to connect to the remote desktop:
apt-get install xtightvncviewer
xtightvncviewer 10.0.0.1:0
Assuming the IP is 10.0.0.1 and the X display is at 0, which it is if it's the only X server running, otherwise use 1.
You can accomplish a similar thing using rdesktop, on the server you install xrdp instead of vnc4server and on the client you install rdesktop (or grdesktop) instead of xtightvncviewer. If you use windows then use whatever rdesktop client it has.