How does a game engine talks to the GPU, what is in the middle of the process?

Question

I want to understand how the commands inside the game engine will be converted into gpu instructions which might use D3D or OpenGL. *I'm starting to learn unity and I would like to understand better what is under the hood. Thanks.

Answer 1

I believe the best answer will be found on a book about game engines, but I can tell you in general terms how things work.

A game engine must keep a world alive. That means, it must keep track of a huge amount of data, like position information of each object, and update its position and other state many times per second. Much of this information like the vertices positions, geometry data and texture is stored on the GPU RAM (VRAM) and kept there, just to be updated when necessary. There are ways to do this efficiently, for example you can add +1 to all vertices of your character, on X coordinate, and the update is done on the GPU side in parallel. You would never want to update vertex per vertex on the CPU side. That is the essence on how to work with real-time graphics.

So CPU and GPU are two worlds that live very close, but do not share their memory space. Data is often moved from CPU to GPU forth and back (mostly towards the GPU). You will want to minimize the amount of data transferred between CPU and GPU, and try to do all the work on the GPU side through shaders.

Typically, in a game engine the CPU handles gameplay commands, AI and most of times physics (GPU can handle physics too). While the GPU renders and recomputes all the effects that are per vertex and per pixel hopefully 60 times per second. Sometimes, data is kept redundantly on both sides to save some communication.

Now, for the ease of development, all this is hidden under layers of scripting language. Scripting language is just understood by the internals of the game engine to perform what I've mentioned earlier. And on top of scripting languages you will have user interfaces with buttons and many other elements to make the job easier to game designers.

Keywords you should look for understanding better a game engine:

1. events
2. logic loop
3. rendering loop
4. shaders
5. collision detection
6. scripting languages for game engines

Answer 2

A Game engine usually does more than just piping down lowlewel rendering commands. You'll need sound and physics, too. Unity provides more than just an abstraction of the rendering device (D3D/OpenGL). It also does the scene management for you for efficient rendering.

Lets say you start loading some assets and place them into the scene. Loading involves creating buffers for vertices/normals/animations on the GPU, textures for materials maybe even shaders. Most liklely you've already heard about scene graphs, where the entities of your scene are managed in. For efficient rendering the objects need to be occluded (entities behind other entities may not be rendered at all since you cant see them anyway) and culled (objects outside the view frustum).

I know this answer is very broad, but hopefully it gives you some ideas, what happens in the background. If you are interested, have a look into the modern rendering pipeline of OpenGL (https://www.opengl.org/wiki/Rendering_Pipeline_Overview). If you just want to write a game though, this is overkill as soon as you got familiar with unity and sorts.