CreateExternalTexture/UpdateExternalTexture issue, iOS.

Hi there-
Edit: I finally got CreateExternalTexture working on iOS with VideoTexturePro if I use BGRA for texture decodes. I want to use YUV for texture decodes.

Alexey, you out there?

I’m using kCVPixelFormatType_420YpCbCr8BiPlanarFullRange format on iOS, which generates 2 textures with CVOpenGLESTextureCacheCreateTextureFromImage-
GL_TEXTURE_2D/GL_RED_EXT, and GL_TEXTURE_2D/GL_RG_EXT. I want to use CreateExternalTexture and UpdateExternalTexture with these two formats, but I can’t see an enum that would match for Texture2D.CreateExternalTexture. Is there a compatible Texture2D format I can use for supporting 2-texture Y + UV decoding? Or am I stuck using kCVPixelFormatType_32BGRA and a single texture?

If I can’t, this is mighty unfortunate, as the whole benefit of using Create/UpdateExternalTexture is to support fast-path decoding. Forcing BGRA decoding causes unnecessarily large performance hits.

Is there any way you could add a bool that lets me flip the texture orientation with Create/UpdateExternalTextures, if you’re drawing with an FBO internally, so my textures dont end up upside down?

Also, is Destroy(nativeTextureFromCreateExternalTexture); necessary when you’re done with it? what exactly is it ‘destroying’ in this case?

BTW: I had previously given up as it had bugs on desktop, (see case 565538 )
Also see http://forum.unity3d.com/threads/117947-OSX-Video-Texture-Pro-Improved-QTKit-Quicktime-video-player-for-MacOSX/page4?p=1520383&viewfull=1#post1520383

Alas, no support (which is unfortunate i know). On the other hand - when you do external texture - we dont really touch texture data. You can actually try doing that: make 2 rgba texture (or alpha lum - which is closer) but create it on top of these yuv gles textures. I think this should just work (but VERY hacky indeed).

i dont get it? the nature of image orientation is related to image layout, so how do expect us to handle it?

well, it should destroy the object. As for killing gl texture - i need to dig code more, but i would say you are good to go (as in - it shouldnt probably touch it)

Edit: just got it goin with YUV using Alpha8 + RGB24 in Unity. Profiled the difference between 1080p h264 decoding in YUV vs BGRA. while there was a measurable difference, it was not a huuuuge difference, so i’m not gonna sweat it. It’s probably easier for other people to deal with RGBA so they don’t have to touch the shader conversion code… but I guess I’ll keep the backend in there in case it is useful in the future for some reason.

Hrm. that’s too bad. Consider this a feature request for more texture enums =p like, for instance, float texture support on OpenGL ES which has existed for many years now…

I wasn’t sure if Create/UpdateExternalTexture were internally rendering to an FBO or some such- in which case you guys could flip the texture so my videos dont end up upside down (which they will and are, if i’m just directly supplying the image from textures created from CVPixelBuffers created from AVPlayerItemVideoOutput) – otherwise, users will have to flip the texture themselves. This is also in addition to a ‘VideoRotationAngle’ (the video track’s preferredTransform) of which I am now supplying, similar to the WebCamTexture API, so that it’s more ‘Unity-like’

That’s good to know.

well, as i said - quickly glancing through code the format shouldnt really matter if you only use textures as shader samplers in unity. Also for FP formats you can currently use no-depth RT (we have fp RT formats) :wink:

well, thats why we have WebCamTexture.videoVerticallyMirrored ;-). The point of this api is actually DO NOTHING behind your back - so yes you should expose this stuff to users

Oh- last i tried, SystemInfo.SupportsRenderTextureFormat was returning false for RenderTextureFormat.RFloat and RGFloat on iOS 6. Maybe it’s changed recently? Or maybe that was a bug? The floating point texture thing was about something else (not video stuff) that required floating point textures- (my workaround ended up being encoding float values in in RG / BA channels and decoding to floats in shaders, because Unity on iOS didn’t support floating point RT’s) but that’s getting a bit off topic.

Anyway, glad to see CreateExternalTexture working on iOS. Now, if we could get RenderTexture support for CreateExternalTexture, even if only on iOS, (and especially be able to write INTO render textures in unity that were created externally), I can imagine a whole nother level of functionality. Imagine if we could create a texture from a CVPixelBufferPool, pass the textureID to Unity, write into it in Unity, and then do whatever else with it in the plugin.

Or CreateExternalTexture support on desktop, cuz it’s totally broken there, still. :-p but you know that…
A guy can dream.