Ever wondered how to create a stunning visual effect reminiscent of a James Bond movie? In this post, we’ll delve into the process of using Visual Compositor to make a stylized music video.
In November 2022, I started a project to produce a video for a short instrumental composition. As I was involved in building a video compositor product in Unity, it was a perfect opportunity to demonstrate the power of this tool.
Step 1. Get something interesting on the screen.
The basic visuals of the scene were created using two character models animated with data from the CMU animation archive. I slowed the animations down to match the song’s tempo and placed the models in the scene, directly in front of the camera.
Step 2. Give it some style.
The models were transformed into silhouettes by utilizing the matte output port of the rendering node. The silhouettes were then blurred and subtracted from the original matte texture, turning them black. Due to the blurred texture adding extra pixel data around the edges of the silhouette, the outlines shifted closer to a white color, creating a sleek outline effect.
These outlines silhouettes are then fed into a motion blur node. The motion blur node works like an accumulator buffer, which interpolates the last frame towards the new frame. It is a simple motion blur which is quite effective with black and white images.
Step 3. Add visuals which match the audio.
The audio visualisation at the bottom of the screen is created by using a custom visual compositor node. The node reads a FFT of the audio stream, and plots it across the bottom of the render texture using a shader. This is then composited on top of a still, public domain photograph of a giant, white stetson hat.
The Really Simple Audio Spectrum Shader
Shader "AudioSpectrum"
{
Properties
{
_Color ("Color", Color) = (0,0,0,1)
}
CGINCLUDE
#include "UnityCG.cginc"
sampler2D _MainTex;
float _AudioSpectrum[1023];
float _SpectrumCenter, _SpectrumScale;
int _BucketCount = 1023;
float4 _Color;
struct appdata
{
float4 vertex : POSITION;
float2 uv : TEXCOORD0;
};
struct v2f
{
float2 uv : TEXCOORD0;
float2 uv2 : TEXCOORD1;
float4 vertex : SV_POSITION;
};
v2f vert (appdata v)
{
v2f o;
o.vertex = UnityObjectToClipPos(v.vertex);
o.uv = v.uv;
o.uv2 = v.uv;
return o;
}
float4 frag(v2f i): SV_Target
{
float s = _AudioSpectrum[i.uv.x*(_BucketCount-1)];
return abs(i.uv.y-_SpectrumCenter) < s*_SpectrumScale ? _Color:0;
}
ENDCG
Subshader
{
Pass
{
ZTest Always
Cull Off
ZWrite Off
CGPROGRAM
#pragma vertex vert
#pragma fragment frag
#pragma target 5.0
ENDCG
}
}
}
Wrap it all up.
Finally, the differet layers are added together to create the final frame. As the video plays (and is recorded), controls within the compositor graph nodes can be adjusted to create interest and emphasize different parts of the soundtrack. It essentially becomes a visual performance tool.
The end result is greater than the sum of its parts. Visual Compositor is a great tool for creating unique, interesting visuals.
Visual Compositor: Visual Compositor Overview | Visual Compositor | 0.30.7-preview
Code Repository: GitHub - simonwittber/music-video