I’m starting work on a 360 VR video player, aimed at mobile devices (iOS/Android Cardboard + GearVR) and I’m evaluating whether it is possible to get video quality from 1080p to 4k in Unity or if I should instead just do the project natively (aka, OpenGL).
I’ve been trying to find information on the state of video in Unity but I haven’t arrived to a clear answer. I’m going to explain the info I’m running on as of now, hopping that someone can add to this or provide an answer or at least show that I am erring on the side of ignorance 
Assume a video in mp4, H264 at 4K resolution (equirectangular mapping):
OpenGL Approach:
If I were to do this in OpenGL, a (somewhat simplified) workflow would entail decoding each video frame, updating a GL_TEXTURE_2D with the pixel contents, and rendering to a sphere with glCullFace(GL_FRONT) as if it where a regular skybox.
With the above setup, the limiting factors would be:
-
Does the GPU in my device support 4K textures? what marketing calls 4k is usually 3840 horizontal, and, an iPhone6 for instance, returns 4096 for
glGetIntegerv(GL_MAX_TEXTURE_SIZE,...). So I’d assume the iPhone6 can load textures as big as required for 4K. -
Fill rate: i.e.: my iPhone6’s GPU might be capable of storing 4K Textures, but it might not be fast enough to fill up a 4K frame buffer at an acceptable frame rate. This unfortunately I think the only way to know is by testing. There is also the added overhead of the Cardboard rendering which means 2 passes per frame.
Unity Approach:
Now for a Unity approach, this is what I did:
- Empty scene with a sphere and cardboard camera at the origin.
- For the sphere, I created a material based on the Unlit shader with one texture and set the cull face to front.
- imported the video and assigned it as the material’s texture.
My observations from the above test are:
UPDATE: I now realize that my test was using Unity’s Move Texture, which is not available on mobile. So if I still pursuit this project in Unity I will anyway have to either do my own native Unity plugin and write the OpenGL code to map the textures to the geometry or purchase an asset like AVPRO
- When importing images, Unity also allows texture sizes of 4096 so 4K should be possible.
- Importing a video however, takes a ridiculous amount of time. I believe this is because Unity actually uses Ogg Theora and so it takes forever to transcode to Ogg. As a matter of fact, sometimes I was able to import and sometimes Unity would just stall.
- When importing a video, I was not given any mmap, filtering or size options as if I was importing an image. So I’m not really sure what Unity is doing to the video. All I get is a quality / compression slider. The end video is still 3840x1920, which was the original resolution.
- The rest of the experiment works ok, as in, yes, the video playbacks in the sphere. But the framerate is not great.
So at this point my conclusions are:
- Is it possible to do 4K in Unity? Kind of, In the sense that you can import 4K videos and reproduce them without any external assets.
- Is it really 4K quality? Probably not. mp4 is lossy and so is ogg, So I assume my video went a second round of lossy compression when it was transcoded to Ogg.
- Is it practical? Not sure. So far, video importing takes an impractical amount of time. Also, 4K in general might be too much for a mobile device rending in stereo, regardless of Unity or doing it straight in OpenGL.
As a side note and to complicate things even more, I was thinking of allowing both downloading and streaming video in the app. But from the above it seems that would be out of the question performance wise. Maybe I have to stick to 1080p or even 720p?