Unreal Engine: accessing/saving in-game images to disk without blocking game thread

Question

I am working on an Unreal based open-source UAV simulation (Microsoft AirSim) where I am trying to capture and save images from a camera that's attached to the drone. The image underneath gives an idea of how the game looks like. The rightmost view on the bottom is the actual view from the camera, the other two are just processed versions of the same image.

Right now the way it's set up in this way: There's a camera asset, which is read through the code as a capture component. The three views in the screenshot are linked to this capture component. The views are streamed without any problem as the drone is flown around in-game. But when it comes to recording screenshots, the current code sets up a TextureRenderTargetResource from this capture component, and subsequently calls ReadPixels and saves that data as an image (please see below for code flow). Using ReadPixels() as it is, is blocking the game thread directly and slowing down the whole game by a lot: drops from ~120 FPS to less than 10 FPS when I start recording.

bool saveImage() {
  USceneCaptureComponent2D* capture = getCaptureComponent(camera_type, true);
  FTextureRenderTargetResource* RenderResource = capture->TextureTarget->GameThread_GetRenderTargetResource();
  width = capture->TextureTarget->GetSurfaceWidth();
  height = capture->TextureTarget->GetSurfaceHeight();

  TArray<FColor> imageColor;
  imageColor.AddUninitialized(width * height);
  RenderResource->ReadPixels(bmp);
}

Looking at this article, it seems evident that ReadPixels() "will block the game thread until the rendering thread has caught up". The article contains sample code for a 'non-blocking' method of reading pixels (through removal of FlushRenderingCommands() and using a RenderCommandFence flag to determine when the task is done), but it doesn't significantly improve the performance: the rate at which images are saved is slightly higher, but the game thread still runs at about only 20 FPS, thus making it really hard to control the UAV. Are there any more efficient asynchronous methods that can achieve what I am trying to do, perhaps, say, in a separate thread? I am also a little confused as to why the code has no trouble streaming those images on-screen as fast as possible, but saving the images seems way more complicated. It's fine even if the images are saved to disk only at 15 Hz or so, as long as it doesn't interfere with the game's native FPS too much.

Answer 1

(...) it seems evident that ReadPixels() "will block the game thread until the rendering thread has caught up"

Using functions that try to access and copy GPU texture data is always a pain. I found that completion time of these functions depends on hardware or driver version. You could try to run your code on different (preferably faster or newer) GPU platform.

Back to code: to bypass single thread issue I would try to copy UTextureRenderTarget2D to UTexture2D using TextureRenderTarget2D::ConstructTexture2D method. When you get your image copied (I hope it take much faster) you could push it to consumer thread at 15 FPS ratio. Thread can be created using FRunnable or Async engine module (using so called Task Graph). Consumer thread could access cloned texture by using texture's PlatformData->Mips[0]->BulkData. Make sure that you are reading texture data between BulkData->Lock(...) and BulkData->Unlock() calls.

I'm not sure, but it could help because your copied texture will be not used by render thread so you could lock it's data without any blocking on render thread. I'm only worried about thread safety of Lock and Unlock operation (didn't found any clue), should be possible unless there are engine RHI calls.

Here is some related code that could be helpful.

Answer 2

You can move the save-to-disk operation from Game thread to other thread.
Please check Async.h for detail

The limitation for thread is that you could not modify/add/delete uobject/uactor in other thread. Multi-thread for UE4

Answer 3

I'm very late to the party, though in case anybody is still having issues with this: I built a non-blocking version inspired by code of AirSim's and UnrealCV's plugins because I had some serious compilation and versioning issues with those.. but needed a smoothly running camera while capturing images myself. This is also based on the async.h class of unreal API provides as the accepted answer suggested. I setup a tutorial repo for this: https://github.com/TimmHess/UnrealImageCapture.

Hoping it may help.

Answer 4

Perhaps you should store screenshots in a TArray until the "game" is done, then process all stored data?

Or look at taking a screenshot from the drone camera instead of manipulating the pixel data? I assume the left and middle drone camera images are material-modified versions of the base camera image.

When you save an image, is it writing an image-format file or a .uasset?

Lastly - and this is a complete shot-in-the-dark - create a custom UActorComponent, add it to the drone, and make it watch for some instruction from the game. When you want to write an image, push the RenderTarget data to the custom UActorComponent. Since actor components can tick on any thread (using BCanTickOnAnyThread=true in the constructor, I think that's the variable name), the actor component's tick can watch for the incoming image data and process it there. I've never done it, but I have made an actor component tick on any thread (mine was checking physics).