Get the output of the main camera and pass it to a native plugin

Questions & Answers
1

Edit: The issue is now resolved. There were a few DirectX errors and mismatches like the texture format that you get from Unity and so fort. Just had to enable the debug flag of the DX device and see what’s really going on.

I am trying to copy (in a efficient manner, not leaving the GPU, etc.) the output of the main camera to a native plugin I am working on. I’ll have to do some shenanigans in there, but for now I’m trying to open a window and render the same picture in the plugin. The issue I’m having is I get only some gray-ish rectangle, and not the actual camera output. Here’s my approach:

In Unity I created a camera that is child to the main camera. I added a script to that camera which makes it render to a texture and passes the texture to my native plugin (based on THE native plugin example):

IEnumerator Start()
    {
        myCamera = gameObject.GetComponent<Camera>();
        if (myCamera != null && myCamera.pixelHeight > 0 && myCamera.pixelWidth > 0)
        {
            CreateTextureAndPassToPlugin();
            yield return StartCoroutine("CallPluginAtEndOfFrames");
        }
    }

private void CreateTextureAndPassToPlugin()
    {
        var myRT = new RenderTexture(myCamera.pixelWidth, myCamera.pixelHeight, 24, RenderTextureFormat.ARGB32);

        myRT.Create();

        myCamera.targetTexture = myRT;
        InitNativePlugin(myRT.GetNativeTexturePtr());
    }

private IEnumerator CallPluginAtEndOfFrames()
    {
        while (true)
        {
            yield return new WaitForEndOfFrame();
			
            GL.IssuePluginEvent(GetRenderEventFunc(), 1);
        }
    }

Then, in my native plugin I get the Unity device and context and map the content of the texture I get from Unity to be able to render it on my window:

IUnityGraphicsD3D11* d3d11 = s_UnityInterfaces->Get<IUnityGraphicsD3D11>();
ID3D11Device* unityDevice = d3d11->GetDevice();

D3D11_TEXTURE2D_DESC uDesc;
unityTex->GetDesc(&uDesc);

ComPtr<ID3D11DeviceContext> unityContext;
unityDevice->GetImmediateContext(&unityContext);

D3D11_TEXTURE2D_DESC copyTexDesc;

copyTexDesc.Width = uDesc.Width;
copyTexDesc.Height = uDesc.Height;
copyTexDesc.MipLevels = 1;
copyTexDesc.ArraySize = 1;
copyTexDesc.Format = DXGI_FORMAT_B8G8R8A8_UNORM;
copyTexDesc.SampleDesc.Count = 1;
copyTexDesc.SampleDesc.Quality = 0;

copyTexDesc.Usage = D3D11_USAGE_STAGING;
copyTexDesc.BindFlags = 0;
copyTexDesc.CPUAccessFlags = D3D11_CPU_ACCESS_READ;
copyTexDesc.MiscFlags = 0;

ComPtr<ID3D11Texture2D> txResult;

hr = unityDevice->CreateTexture2D(&copyTexDesc, nullptr, &txResult);

D3D11_BOX box;
box.left = 0;
box.right = uDesc.Width;
box.top = 0;
box.bottom = uDesc.Height;
box.front = 0;
box.back = 1;

unityContext->CopySubresourceRegion(txResult, 0, 0, 0, 0, unityTex, 0, &box);

D3D11_MAPPED_SUBRESOURCE unityMapped;

hr = unityContext->Map(txResult, 0, D3D11_MAP_READ, 0, &unityMapped);

For some reason I end up getting only gray from the pixels in unityMapped.pData (see below).

65044-output.png
65044-output.png1184×525 3.59 KB
Do you think my approach is correct? Any ideas? Thanks!

2

do you get solution?

3

Yes. It’s working now. There were a few DirectX errors and mismatches like the texture format that you get from Unity and so fort. Just had to enable the debug flag of the DX device and see what’s really going on.