I’m trying to integrate OpenCV with ARKit and I can’t figure out how to access the captured image from ARKit.
I was hoping that in the ARFrameUpdatedEvent delegate I’d be able to access ARFrame and presumably the captured image - which I could then pipe into OpenCV, but this only returns a UnityARCamera object that doesn’t seem to expose ARFrame or the captured frame image in any way.
Can anyone point me to a way I might accomplish this? Is it possible?
In UnityARCamera, look at videoParams.cvPixelBufferPtr - might be what you need (its void* pointer to the capturedImage pixelbuffer).
@jimmya Do you have an example of this working? I’m trying to use Texture2D.CreateExternalTexture with cvPixelBufferPtr and the resultant texture is always white. RGBA: .804, .804, .804 .804. Is there perhaps a timing issue with accessing this pointer? Are the contents only valid at certain times or something? Or maybe I’m using the wrong format to create this texture? ( TextureFormat.ARGB32)
In fact…from checking out the YUVMaterial shader, I don’t think there is a valid camera frame texture anywhere? It seems like the fragment shader combines the Y and CbCr textures and spits them out to the screen. But this video frame isn’t actually stored anywhere…?
Ok–so the answer is…make a new camera, put the UnityARVideo behavior on it, have the culling flags set to it doesn’t render anything, have this camera output to a RenderTexture. That RenderTexture will include only the video frame! Works great!
Hi,
Can you provide sample code or the steps on how to set the culling flags not to render anything and how to create a RenderTexture on the camera? Thanks
I think the problem you’re having is related to the fact that cvPixelBufferPtr is a pointer to a native CVPixelBuffer object, which wraps a pixel buffer and not a native texture. Therefore, you’re getting garbage.
If you want to extract the actual buffer, you would need to write some native iOS code I think. Have a look at CVPixelBufferLockBaseAddress and then CVPixelBufferGetBaseAddress (possibly per plane if it’s YUV420).
Any new insights concerning this issue? There are actually a few developers that would like to use the camptured ARKit frame in OpenCV. By using the Rendertexture approach it is actually hard to get a decent frame rate due to the necessary conversion from Rendertexture to Texture2D (which is needed in order to be converted to a Mat object).
You should be able to use a Rendertexture directly in a material. Notice a material uses a Texture, and a RenderTexture is a Texture.
Oh I think we have a misunderstanding here, I am talking about the Mat class of Open CV which is basically a matrix that contains image data and not the texture for a Unity Material.
You can extract pixel values from the RenderTexture and pass it to the OpenCV mat, but it is an expensive operation. Would be great if there was a cheap (as cheap as possible) way of doing this as AR has loads of places where you want to do custom computer vision operations, on top of those provided. The suggestion I have seen is to extract the pixel values natively, but it feels like rather than each of us doing this separately it should be part of some core functionality (pretty please! 
)
If you look in code how the arkit remote sends video frames across to editor, there is a way to do it there.
Thanks for the hint, I’ve tried to use the ARKit Remote connection code, but it nonetheless relies on the Texture2D and the Apply function and therefore results in immense FPS drops. I guess the only viable option that remains is writing native code or waiting for an update where this feature is implemented.
If I am fine with R8 VideoTextureY from UnityARVideo, how to convert it (it’s a CreateExternalTexture) into Unity Texture2D so I can access pixels? I tried accessing with IntPtr but I am only getting garbage. 
I am trying to avoid RenderTexture solution mentioned above.
+1 to have it exposed. In ArCore they use System.Runtime.InteropServices.Marshal.Copy and it works great!
As I mentioned before, I had put in a fastpath for this in the plugin called “SetCapturePixelData” - you send it some byte arrays from C# and it fills it in native code. And you can double buffer the byte arrays you send it so that there is less sync problems. All this is done in https://bitbucket.org/Unity-Technologies/unity-arkit-plugin/src/56302bff1b8adc2190c3c1b7db2ab015cd79dc8e/Assets/UnityARKitPlugin/ARKitRemote/UnityRemoteVideo.cs?at=default
Thanks! It works great! You are awesome!
Apologies if this is a noob question - How do extract the image from the separate Y and UV texture byte arrays? Ideally I would like a png encoded image bytes. Any pointers are appreciated on this.
The current workflow seems so - > Y, UV arrays passed to a shader
Shader converts pixel by pixel
const float4x4 ycbcrToRGBTransform = float4x4(
float4(1.0, +0.0000, +1.4020, -0.7010),
float4(1.0, -0.3441, -0.7141, +0.5291),
float4(1.0, +1.7720, +0.0000, -0.8860),
float4(0.0, +0.0000, +0.0000, +1.0000)
);
float2 texcoord = i.texcoord;
float y = tex2D(_textureY, texcoord).r;
float4 ycbcr = float4(y, tex2D(_textureCbCr, texcoord).rg, 1.0);
rgba = multiply(ycbcrToRGBTransform,ycbcr);
Any better method to abstract this and write it more clean and easy?
The only other way I know is to use render texture from second camera and read from there. I don’t think there is a way today to get RGB image from AR Core or AR Kit.
If I use the render texture, I lose the video background when using ARKitRemote Also there seems to be a delay in the render if I apply it to quad and re render it again on the plane.
Is there any way to capture the image on click of button while the video keeps rendering on the screen? Any pointers on this?
I am not exactly sure how to add the second camera. I tried that and I get errors that I have two listeners etc.,
Did you ever figure out a good way to get RGB from this?