How to write to render buffer in Plugin?

Unity Engine
,
1

I’ve subclassed UnityAppController, and registered my custom render delegate. I’m trying to write to the (Metal) frame buffer in onFrameResolved. While it runs, and the logs spit out sensible data, I can’t seem to change the final image rendered by Unity. Is systemColorRB the correct buffer I should be writing to? Should I be writing to it in a different way than replaceRegion?

#import "RenderDelegateTest.h" // nothing of note in the header

#import <Metal/Metal.h> //I'm confused why this wasn't included in an upstream(?) header

@implementation RenderDelegateTest

UnityDisplaySurfaceMTL *metalDisplaySurface;
int bufferBytesPerRow;
void *bufferBytes;
int bufferBytesPerImage;

- (void)mainDisplayInited:(struct UnityDisplaySurfaceBase*)surface
{
    metalDisplaySurface = (UnityDisplaySurfaceMTL*)surface;
}

-(void)checkBuffer:(MTLTextureRef)buffer
{
    int _bufferBytesPerRow = [buffer width] * 4; //MTLPixelFormatBGRA8Unorm = 80
    int _bufferBytesPerImage = _bufferBytesPerRow * [buffer height];
  
    if (bufferBytesPerImage != _bufferBytesPerImage)
    {
        bufferBytesPerRow = _bufferBytesPerRow;
        bufferBytesPerImage = _bufferBytesPerImage;
  
        //allocate some buffer we can copy back and forth between
        bufferBytes = malloc(bufferBytesPerImage);
    }
}

-(void) onFrameResolved
{
    MTLTextureRef buffer = metalDisplaySurface->systemColorRB;
  
    [self checkBuffer:buffer];
  
    MTLRegion wholeRegion = MTLRegionMake2D(0, 0, [buffer width], [buffer height]);
  
    //copy the buffer data into a local array
    [buffer getBytes:bufferBytes
        bytesPerRow:bufferBytesPerRow
        fromRegion:wholeRegion
        mipmapLevel:0];
  
    unsigned char *_bufferPtr = (unsigned char*)bufferBytes;
    for (int p = 0; p < 10; p++)
    {
        //test to see if there is colour data there (there is!)
        NSLog(@"\t\t\t %i", _bufferPtr[0]);
        _bufferPtr++;
    }
    for (int i = 0; i < 3000; i++)
    {
        //set a bunch of pixels to see if we can affect the buffer.
        _bufferPtr[0] = 255;
        _bufferPtr[1] = 200;
        _bufferPtr[3] = 10;
        _bufferPtr[4] = 255;
      
        _bufferPtr++;
    }
  
    //copy the buffer data back
    [buffer replaceRegion:wholeRegion mipmapLevel:0 withBytes:bufferBytes bytesPerRow:bufferBytesPerRow];
}

@end
2

while i am not sure that updating data like that is what you really want, the issue is with “syncing” - you try to do stuff immediately (while actual rendering happens in parallel or might not even start yet).
what you should do is get id by calling UnityGetCommandBufferMetal() and then call commit and waitUntilCompleted. Two caveats:

  1. i am not sure how unity would react to this (well, we expect to commit command buffer ourselves)
  2. it will be DEAD slow (because you would be waiting for gpu to finish)
    another option would be to addCompletedHandler for this command buffer and do this magic here (should actually get rid of both issues). Though at this point you cannot tweak the actual buffer for current frame - but you can do magic like reading and do smth (e.g. you can copy contents and do smth with them)
    and third option would be to rethink what you are trying to do - why would you do anything with final buffer on cpu? that kinda doesnt make sense