Read the contents of a RenderTexture in DX11 within a native plugin

I’m trying to move the data from a RenderTexture to the CPU as fast as possible within my Native Plugin.
I tried using ReadPixels and GCHandle to move the data and that works but it is far too slow.
As such I figured I would implement it manually for all the different rendering API’s within my native plugin.
I am however struggling to do so for DX11.
I can take the NativeTexturePtr in Unity and cast it to a ID3D11Texture2D and read the desc but as soon as I try to read the actual data from it or map it to a subresource Unity will just crash without generating a report.
This is the code in my native plugin function:

ID3D11DeviceContext* ctx = NULL;
m_Device->GetImmediateContext(&ctx);

ID3D11Texture2D* lDestImage = (ID3D11Texture2D*)textureHandle;
D3D11_TEXTURE2D_DESC desc;
lDestImage->GetDesc(&desc);
D3D11_MAPPED_SUBRESOURCE resource;
UINT subresource = D3D11CalcSubresource(0, 0, 0);

ctx->Map(lDestImage, subresource, D3D11_MAP_READ_WRITE, 0, &resource);

Hopefully someone here is a DX11 wizard that can help me out because I’ve been banging my head at this for about a week now.