Hi everyone!
I finally took a bit of time to produce a simple 360 VideoPlayer example project with instructions.
For the Impatient
- Download the Unity package here: https://oc.unity3d.com/index.php/s/kceb0rAE9fDcdqE
- Import it into Unity using Assets → Import Package → Custom Package …
- Open the Scene.
- Hit play.
- Move the mouse to look around.
In 5.6, the VideoPlayer doesn’t offer workflow enhancers to streamline the usage of 360 videos so everything must be done by hand. There are a handful of steps to do manually which are all explained below with enough details for you to understand exactly what is happening and why.
What’s a 360 Video?
What we call a 360 video can take many forms. One of the most common forms is a movie with a 2:1 aspect ratio, often 3840x1920, that uses an equirectangular projection to unwrap a sphere into a rectangle, as described here: Equirectangular projection - Wikipedia
When looked at without putting them back into a sphere, these videos look deformed, especially at the top and bottom, corresponding to the sphere’s poles.
Creating a Projection Screen Inside Unity
As mentioned above, in order for the 360 video to look right, it must be viewed when applied to the inside of a sphere. This can be done in Unity but typically poses two - solvable - problems:
- One cannot directly apply a texture to the inside of a sphere
- Once overcome with some normal-inversion tricks, applying a texture inside a sphere leaves visible artifacts at the poles.
Both of these problems can be overcome with a relatively simple shader, see the details further down. So our projection “room” will be composed of a camera, sitting at the center of a sphere that uses this shader.
Looking Around
Since our projection screen wraps the camera completely, the camera can only look at a small portion of its inside surface at any given time. The demo project includes a script that lets you control the camera orientation using the mouse.
Future VR integration might automate this so one doesn’t have to add these inspection tools to the scene per se. This is conceptually similar to the Scale control at the top of the Game View.
Playing The Video
Once you have the camera, with its mouse control, and sphere-screen prepared, playing back a video in this scene is just a matter of importing a 360 video into Unity and drag-and-dropping the resulting Video Clip onto the sphere.
This drag-and-drop is a workflow helper that will automate for you:
- The creation of the Video Player
- The initialization of the Video Player’s Render Mode to “Material”
- The selection of the
_MainTex
texture parameter in the first material of the sphere’s renderer as the target for the video playback.
Future VR integration in the Video Player might augment the offered Render Mode options to recognize that the video is 360 footage and add options for the interaction with the camera such as projection type, field of view, etc. It might also free you from having to create a sphere and render directly into the scene.
Doing It From Scratch
The Unity package downloaded earlier has a working example of all the pieces assembled together. But if you want to redo each step for yourself to better understand how everything interacts, here is a step-by-step list of operations:
- Create a new scene
- Copy the 360 equirectangular shader (360Equirectangular.shader) into the Assets folder. Have a look inside to familiarize with the simple math involved.
- Create a new Material named InsideSphere.
- Change the InsideSphere shader to 360/Equirectangular so it uses the shader you’ve imported earlier.
- Create a Sphere using the GameObject->3D Object->Sphere menu.
- In the Sphere’s Mesh Renderer, set the material to use the InsideSphere material created above.
- Set the Main Camera Position to 0, 0, 0 so it sits in the center of the sphere.
- Copy the MouseFollow.cs script into your Assets folder.
- Drag-and-drop the MouseFollow script onto the Main Camera so the mouse movements will control the camera orientation.
- Copy the 360_test_foggy_park_001.mp4 video into the Assets folder.
- Drag and drop the Video Clip onto the Sphere in the Hierarchy view. This will add a VideoPlayer component to the Sphere with the clip already set as its source, and that targets the current game object’s renderer.
If the 360 video clip has audio in it and you want to hear it:
- Set the VideoPlayer’s Audio Output Mode to Audio Source
- Add an AudioSource component on the sphere
- Drag and drop the Audio Source (by clicking on the Audio Source title label) into the Audio Source field in the VideoPlayer editor.
The Equirectangular Shader
The shader used to texture the video into the sphere (360Equirectangular.shader
) uses a cartesian-to-spherical coordinate conversion as described here:
The fragment’s normal is used, instead of its coordinates, in order to produce texture coordinates, which is exactly the inverse operation that was done when transfering the pixels from the 360 camera into a rectangle frame. The calculation is simplified by normalizing the sphere radius to 1 since ultimately we only want the inclination and azimuth angles to convert them back to [0, 1] values to provide texture sampling coordinates. This saves a few divisions in the calculation.
The trick that performs the normal inversion hinted at earlier in this post is taken care of by the Cull Front directive at the top of the shader pass. This is one way to have the objects rendered “inside out” as described here: Unity - Manual: ShaderLab: commands
Final Note
I’ll probably convert this into a blog post when Unity 5.6 is officially released. Feel free to make suggestions on how this could be improved to be clearer or more useful.
Dominique
A/V developer at Unity