Shader replacement in VR. Now and the future.

Unity Engine
1

I know starting out that shader replacement and VR, in general, are not an ideal combination. VR necessitates high frame rates and shader replacement is expensive. But for what I’m trying to accomplish there is no alternative, I need shader replacement.

Issues with current shader replacement in VR
Currently, RenderWithShader() / manually rendering disabled cameras don’t support single-pass stereo, which makes it painful enough, as each needs shader replacement performed individually. I’m now trying to get multipass shader replacement to work while VR cameras are active and are at a loss.

Even if the disabled, replacement camera is in no way connected to the VR camera (ie. not using Camera.CopyFrom(myVRCamera) and the replacement is not rendered during the VR cameras pre-render/post-render) the replacement camera either only seems capable of rendering to a single target at a time (SinglePass) or the first target of the shader replacement is blitted into the left eye of the VR headset seemingly arbitrarily (MultiPass) when rendering to multiple targets.

In my scenario, this means performing 3 separate passes of shader replacement, per eye, which makes my current extension understandably unusable in a lot of VR use-cases.

Separate bug reports have been submitted for both the single-pass and multipass issues here :
SinglePass - (Will edit to amend if opened/accepted)
MultiPass - (Will edit to amend if opened/accepted)

I’ve attached the project submitted to both bug reports if anyone else is interested.

(Side note) - Issues with frame debugger in VR
As an aside, the frame bugger continues to update with the HMD’s orientation as you use it. While this may sound like correct behavior, in reality, it means what is currently being culled is constantly changing as your trying to navigate the frame debugger. This makes it near impossible to use without putting your headset down, as the position in the frame queue is constantly changing with the varying number of passes. So why not just freeze the HMD position when the frame debugger is enabled?

I’ve not submitted a bug report for this as it could questionably be conceived as the correct behavior, instead, I’m just making an argument for changing the behavior or adding an option.

(Can skip to here if you like)
So my question is not can you fix these issues (Though that would be nice) but rather, are you planning on supporting the current model of shader replacement in the future? Or will you be deprecating it in favor of Scriptable Rendering Pipelines, if and when they come out?

Also if scriptable render pipelines are still quite far out (Currently listed as “In-progress, timelines long or uncertain”), are there any known workarounds?

Cheers
Dan

3244885–249588–VR Test.zip (3.88 MB)

2

So I don’t have any real suggestions for fixing any of these issues, but I do have one idea for working around the multi-target problem.

Don’t use multiple passes. Use a single render target.

Obviously when you’re trying to pack depth, normals, and lets say a mask into a single texture is that’s a 24 bit float depth, an 8 bit per channel or better normal, and at least 4 mask channels, that’s at least 8 texture channels!

But it isn’t really, that’s just 24+24+4 bits, or 52 bits you need. You could pack all of that into a single RGFloat (64bit) texture. Even if you wanted to use 10 bits per component normals that would be 58 bits, which still fits in that RGFloat, and you don’t really need the normal’s z component in that case just the sign, so it’s only 49 bits!.

If you’re thinking this would need bitwise operations, they help, but there’s ways around this that work even for GLES 2.0! (Though it should be noted GLES 2.0 devices don’t support the RGFloat format.)

1 Like
3

@bgolus - That’s an awesome idea! I have different methods for non-VR devices so, assuming DirectX11, 12 and Vulkan all support RGFloat I’m in! I was not aware there was a safe and precise way to do bitwise operations/pack textures in floats, I could use this in so many places!

The depth can fit int the R channel pretty easily, wouldn’t even have to pack it. so it should be quite easy to pack 24 bits into a float, then the sign.

I’ll admit it’s still a little beyond me currently, but it looks like fun to learn. :slight_smile:
Thanks!!

4

The nice side effect of this is having only a single texture to sample for all of the data.

I do wonder why you’re not just using the built in depth and normal buffers though. These work properly for VR and are likely already being generated (at least the depth is) even for forward rendering on desktop and console as its part of how Unity does real time directional shadows and soft particles. The mask is the only thing you really need to render on your own.

5

@bgolus - The depth is likely used but not always, so I’d rather not assume it’s being used and have some customers render an additional pass on everything. And in deferred rendering, it’s not “resolved” until after the Deferred rendering has drawn everything into its buffers. I need it while drawing my decals into these buffers, not afterward (Do have customers that want to use Deferred rendering with VR, so it needs to be considered).

Using the depth-normals texture only gives 16bit depth precision and 8bit normal precision, which causes “banding/dancing”(Unless you set the far clipping plane really close, which is not practical), and normal artifacts.
You can get away with these on old mobile phones, but on computer screens it’s pretty noticeable. In VR it’s really noticeable.

6

It works perfectly! Thank you! What an awesome idea. For anyone else who’s interested I’ll post the relevant code used to pack an unpack. It likely won’t apply to your specific situation, but between it and this post, you should be able to pick up the common patterns and apply it to whatever you require.

Packing (Packs 10bit per channel normal and 4 mask layers into float)

float PackNormalMask(float3 Normal, float4 Mask)
{
    //Encode normal (0 - 1)
    float2 normal = EncodeViewNormalStereo(Normal);

    //Pack into float
    uint packed;
    packed = ((uint)(normal.x * 1023.0 + 0.5)); //Bits 0 to 9 (MMMMMMMMMM)
    packed |= ((uint)(normal.y * 1023.0 + 0.5)) << 10; //Bits 10 to 19 (MMMMMMMMMM)
    packed |= ((uint)(Mask.x + 0.5)) << 20; //Bit 20 (M)
    packed |= ((uint)(Mask.y + 0.5)) << 21; //Bit 21 (M)
    packed |= ((uint)(Mask.z + 0.5)) << 22; //Bit 22 (M)
    packed |= ((uint)(Mask.w + 1.5)) << 23; //Bit 23 (E), To prevent an exponent that is 0 we add 1.0

    return (float)packed;
}

Unpacking (Seperates out float and unpacks into normal and mask vectors)

//Grab packed texture
float2 packed = tex2D(_CustomDepthNormalMaskTexture, ScreenPos).xy;

//Output depth
Depth = packed.r;

uint NormalMask = (uint)packed.g;

//Unpack & decode viewspace normals
float x = ((((NormalMask) & 0x3FF) / 1023.0));
float y = ((((NormalMask >> 10) & 0x3FF) / 1023.0));
float3 surfaceNormal = DecodeViewNormalStereo(float4(x, y, 0, 0));

//Unpack mask
Mask.x = (NormalMask >> 20) & 0x1;
Mask.y = (NormalMask >> 21) & 0x1;
Mask.z = (NormalMask >> 22) & 0x1;
Mask.w = ((NormalMask >> 23) - 1) & 0x1;
2 Likes