Recording Textures

Just One More Step
For some more advanced use cases, you might want to record video frames from a Texture in Unity. Typically, recording from a Texture requires you to first read the pixel data from the GPU before committing the data to the recorder.
Unity provides two ways to readback pixel data from a Texture into system memory. They differ based on whether the client is willing to accept a deferred completion of the readback.
Synchronous Readbacks
A synchronous readback will request pixel data from the GPU and block until the request has been completed.
// Say we have some `RenderTexture`
RenderTexture renderTexture = ...;
var width = renderTexture.width;
var height = renderTexture.height;
// We can perform a synchronous readback using a `Texture2D`
var readbackTexture = new Texture2D(width, height);
RenderTexture.active = renderTexture;
readbackTexture.ReadPixels(new Rect(0, 0, width, height), 0, 0);
RenderTexture.active = null;
Once the above code executes, the pixel data will be accessible through the Texture2D.
// Commit the pixel buffer
recorder.CommitFrame(readbackTexture.GetPixels32());
Due to the concurrent nature of GPU's, such a readback will typically result in a pipeline stall. This means that both the CPU and GPU cannot do any other work until the request is complete. Pipeline stalls are typically very expensive, and will cause a very noticeable frame rate hit in most cases.
Though expensive, synchronous readbacks exhibit very predictable memory consumption patterns.
Asynchronous Readbacks
Unity provides the AsyncGPUReadback API for performing an asynchronous readback.
// Issue a readback request
AsyncGPUReadback.Request(renderTexture, 0, request => {
    // Once complete, access the data container
    var nativeArray = request.GetData<byte>()
    // And commit the pixel buffer
    recorder.CommitFrame(nativeArray.ToArray());
});
This readback will request the data from the GPU but will not wait for the GPU to complete the request. When the transfer is complete, Unity will invoke the provided callback. We can then commit the pixel buffer to the recorder within this callback.
The advantage with this approach is that it provides much better performance over synchronous readbacks. There are a few disadvantages with this approach:
It adds latency on the order of a few frames
It has less predictable memory consumption patterns than synchronous readbacks.
It is not supported on all platforms and devices, so you must check at runtime.
​CameraInput and ScreenInput will default to asynchronous readbacks on devices that support it.
Texture Inputs
NatCorder provides primitives that implement both synchronous and asynchronous readbacks, so that you don't have to. The TextureInput class handles synchronous readbacks:
// Create a texture input
var recorder = ...;
var clock = ...;
var textureInput = new TextureInput(recorder);
// Commit video frames from a texture
var renderTexture = RenderTexture.GetTemporary(...);
textureInput.CommitFrame(renderTexture, clock.timestamp);
Similarly for asynchronous readbacks, NatCorder provides AsyncTextureInput:
// Create a texture input
var textureInput = new AsyncTextureInput(recorder);
// Commit video frames from a texture
var renderTexture = RenderTexture.GetTemporary(...);
textureInput.CommitFrame(renderTexture, clock.timestamp);
Workflows - Previous
Recorder Inputs
Next - Workflows
Offline Recording
Last modified 10mo ago