Stereo-friendly Outline Shader for VR difficulties

Graphics Shaders & Materials
1

I’m trying to get a black border outline shader to work in VR, but I’m having difficulties with stereo separation.
The border outline is created by extruding out the normals of the mesh faces.

VertexOutput vert (VertexInput v) {
    VertexOutput o = (VertexOutput)0;
    o.uv0 = v.texcoord0;
    float width = (o.uv0.r*0.02);
    o.pos = mul(UNITY_MATRIX_MVP, float4(v.vertex.xyz + v.normal*width,0));
    return o;
}

This code works perfectly fine with a single camera, but creates problems when split into stereo for VR.
Whether or not the bordered outline of a face is visible depends on the viewing angle of the camera.
Since VR has two cameras, there exists a position where an outline is visible in one eye, but not the other. This makes long faces especially painful to look at when the Camera is in just the wrong position.

I’ve tried making my own MVP matrix with a V that comes from camera.projectionMatrix, but I’m not sure if this approach is correct, nor sure if I’m even implementing the approach correctly in the first place. This is my first time trying to code a shader, and I definitely think it’s out of reach given current abilities.

Any help would be appreciated!
Thanks

2

I’ve used the same approach with VR and haven’t noticed any issues with it. In my case the extrusion is fixed or based on the distance to the camera to get a uniform thickness. Why are you using the first coordinate of the uv to determine the thickness? Is the model you apply it on a closed model?

3

The shader originally had an additional texture mask that controlled the thickness of lines. I accidentally left in the UV portion of that (I only half know what I’m doing). And I think the models I will apply the shader to will be closed, yes.

Would you be okay with posting the code for one of your VR shaders that does that? And the C# script too, just to be safe? I’ve experimented a lot, but still can’t seem to figure out where I’m going wrong with it. I would be very grateful!

4

Well, as long as the model is closed, it should work fine. The potential issues I can see are very minimal. You might want to remove the thickness control and normalize the normals:

o.pos = mul(UNITY_MATRIX_MVP, float4(v.vertex.xyz + normalize(v.normal)*_Width,0));

Where _Width is a shader property instead of a thickness based on the uv’s.

5

That’s basically the same as my code, and it’s not a working solution. If you want to see why it doesn’t work, apply it to a cube, or any object with a long straight face. Get close to one of the edges with an outline and slowly move your head to the left or right until it the outline disappears. You’ll notice it disappears in one eye before the other, making it very uncomfortable to look at. The MVP matrix is different for each eye because the V matrix depends on the camera view, because the eyes are separated into stereo. This allows faces to be culled in one eye, and not the other. Unless you’re rendering in mono, or turning on an option to make the MVP matrix behave differently in VR that I don’t know about, this solution doesn’t solve the problem.

I’ve still been trying, but without luck. Now I’m just crossing my fingers and hoping Unity introduces some sort of VR Mono MVP in shader code, and I’ve given up for the time being.

6

Ah, I think I get the issue now. I’m pretty sure my outline part has backface culling turned off. That should do the trick.

So, my outline works by first rendering the object normally while also marking the stencil buffer. Then I render the outline without backface culling and with a solid color where the stencil buffer is not marked.

Just did a check and backface culling is indeed off in the second pass.

7

Sorry, I didn’t mean that culling gets turned off for the entire second pass. I meant that on a per-face basis, the culling is different in each eye given the viewing angle. I’m just using Cull Front. I haven’t tried using a stencil though (I didn’t even know they existed), so I might give that a go later. Thanks for the tip!

8

Yes, I understand, I’m talking about the same thing. You’re rendering inverted (Cull Front), which doesn’t need the stencil buffer. I’m rendering 2-sided (Cull None) with the stencil buffer to reject pixels that go over the object itself. That should solve your issue.