iOS using AVCaptureVideoPreviewLayer to get faster camera for augmented reality

Unity Engine
1

I’m doing an AR app, with the ability to record and playback the Augmented experience.

I’m not having fun with the webCamTexture on any of my iOS devices - it’s simply too slow.

It’s almost okay, but it’ll never be good enough to record with (too many artifacts when moving quickly).

Apple’s AVCaptureVideoPreviewLayer has no such problem.

I was using cocos3d before, and was very succesfully putting a AVCaptureVideoPreviewLayer behind the cocos3d gles view.

I’ve just been tinkering around with unity3d’s iOS project and didn’t have much joy getting it to do the same thing…

Has anyone succesfully made unity3d’s view transparent, and put another uiview below it?

2

in 4.2 (soon to come) we’ve added CVOpenGLESTextureCache wrapper to trampoline, and the first thing to be implemented with that was webcam texture. we have added WebCamTexture.MarkNonReadable() (ios only) - the effect of it is that we can direct video feed directly to texture (using this cv texture cache). And yes, it is very fast :wink:

3

I’d like to bump this thread. My studio is working on a cross platform title and I was recently tasked to prototype in Unity to see if we should get a license instead of writing separate codebases. Is it possible to toggle WebCamTexture.MarkNonReadable() on and off after the texture is created? I notice an enormous performance increase with MarkNonReadable turned on. Our software only needs to read single images from the camera periodically (no video processing), but it also requires a live view to be present.

4

it is doable, but not implemented right now. Actually most of the related code is exposed in trampoline so you can just go and do it yourself (hides) ;-).

5

I was just actually able to get around it now by using a camera in front of the WebCamTexture and snapping it to a RenderTexture when I need the frame. Had to think more virtually-physically I guess! Trampoline is the Xcode Project that gets created I see, and yeah I see this code in there. I’m tempted but I’ll stay away considering the fact that it might change down the line, which would defeat the point of using Unity for us (hides).

Regardless I think I’ll be messing around with that sometime in the future to be able to access some native features like bringing up a share window or something.

There is another bug… that may not be tracked… In OpenGL ES 3.0 I was sometimes getting 0x0500 GL_INVALID_ENUM errors from GlesHelper.mm while trying things out with the WebCamTexture, but they seem to have mysteriously disappeared now. I think calling GetPixels32 from a WebCamTexture is what triggered it. (Regardless of MarkNonReadable.)