The v
in xvfb
means that you are using a virtual display.
You probably want to look into something like x11vnc.
x11vnc allows one to view remotely and interact with real X displays (i.e. a display corresponding to a physical monitor, keyboard, and mouse) with any VNC viewer. In this way it plays the role for Unix/X11 that WinVNC plays for Windows.
AskUbuntu has the guide for setup as such (quoted below)
You'll need xrdp together with x11vnc if you want remote access to your Unity desktop.
First, install xrdp and x11vnc: apt-get install xrdp x11vnc
(or whatever your distribution equivalent is)
Then edit /etc/xrdp/xrdp.ini
and make sure it contains at least the
following:
[globals]
bitmap_cache=yes
bitmap_compression=yes
port=3389
crypt_level=high
channel_code=1
max_bpp=24
[xrdp1]
name=Remote Desktop
lib=libvnc.so
username=ask
password=ask
ip=127.0.0.1
port=5900
Restart xrdp for the changes to take effect:
/etc/init.d/xrdp restart
Now open up a terminal window (Ctrl+Alt+T) and set up a VNC password
(you don't want to leave wide open access to your desktop from the
internet):
x11vnc -usepw
You'll be prompted to set up a password, that will be used for RDP
authentication.
The last step is to start x11vnc. There are tons of options that you
can use, but here is an example to get you started (run this in the
same terminal window you just opened):
x11vnc -rfbauth ~/.vnc/passwd -display WAIT:0 -reopen -nevershared
-forever -wirecopyrect always -scrollcopyrect always -norepeat -1 -xkb -capslock -skip_keycodes 187,188 -bg -o ~/x11vnc.log &
The & at the end will leave x11vnc running the background, so you can
close the terminal window and leave it running.
You'll need to add this command to your startup scripts if you want
x11vnc to run automatically after a reboot.