To summarize my understanding of your situation: you need to play HD videos in a virtualized guest system acting as client to an X Server located in another virtualized guest.
Your question is mainly to which guest you should dedicate you GPU in order to lower the CPU load as much as possible and have a general overview how the display will actually work.
Most chances are that the GPU will have to be handled by the guest running the X Server. I indeed see no way how a client would send video frames directly to the graphic card without knowing anything about the containing window handled on server side.
For information, you may also want to consider using Linux framebuffer which can allow you to play videos right from the command prompt without X at all.
As to understand how it works, the trick here is that there is not a single way to play videos in an X environment, and depending on your platform details, some may be faster, some may show glitches with certain video files, some may not work at all.
I recommend you to use VLC media player to proceed with your tests. It is available in most Linux distributions repositories and allows you to manually select which way it will use to display the video (from the menu, go in Tools > Preferences > Video, then select you choice in the Output drop-down widget).
Whatever way is used, it will always generate a lot of traffic between the X client (the video player) and the X Server, so if both virtualized guests are running on the same host you will most likely prefer to ensure that there is no encryption occurring on their exchange. In other words, you may most likely leave SSH-tunneling out and prefer to use an old-fashioned X display redirection:
On the server host, ensure that the X server listens on an external interface (the exact configuration step is distribution dependent, if manually modifying a file ensure it will not get overwritten by some automatic configuration, you may also need to adapt your firewall rules) and allow incoming X connections from the guest environment using the xhost command:
$ xhost +<XCLIENT_IP>
On the client host, export the $DISPLAY
variable before launching VLC:
$ export DISPLAY=<XSERVER_IP>:0.0
$ vlc
Certain output modes will make VLC to go through the X stack, while some other will make VLC to circumvent it and directly contact the underlying video hardware through some library. In your situation, only the former will work. Trying to use the wrong output type will make VLC to either display no video (while still playing sound) or crash when opening the file.
Best candidates will therefore be Xvideo (an X extension dedicated to video playback, it replaces the older but still available X11 output), GLX and OpenGL (both make VLC to rely on a API interfacing OpenGL and X display system).