toon outline with DOF?

Unity Engine
1

Hi everyone,

i have a little issue with toon outline shader and depth of field. when dof is on, my toon shaded objects outline gets blurred even though it’s on the foreground and in focus… is there a way around this? i tried using double cameras and having character on one and the environment on another, but its not really my problems solution…
anyone can offer help?

thanks a lot…

2

i attached a screencap of my problem…
http://img199.imageshack.us/i/outlinedof.jpg/
:face_with_spiral_eyes:
any help? anyone?

3

yep, the problem is that outlines don’t have a depth value, so the DOF will just defocus based on the original depth value. quite some legword should be needed here. you could e.g. modify the DOF and EdgeDetect shaders to accomplish a better look:

  1. store an outline mask (result of edge detection filter) along with the minimum depth value from the depth comparison check in a texture
  2. after calculating the circle of confusion (the first step in the DOF effect), update the COC based on the depth vales from the texture in 1) for all outlines
  3. continue DOF as usual

this might totally not work at all, but is all i can spontaneously think of.

4

what about making an outline by duplicating the mesh and inverting the normals? that’d have DOF, right?

5

It should but it doesn’t, anyone has any idea?
The toon outline shader has ZWrite on so it should write to depth.

    SubShader {       
        LOD 900       
        UsePass "Toon/Basic/BASE"
       
        Pass {   
            Tags { "Queue"="Geometry-1" "RenderType"="Opaque" }
            Name "OUTLINE"
            Tags { "LightMode" = "ForwardBase" }
            Cull Front
            ZWrite On

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
            half4 frag(v2f i) :COLOR { return _OutlineColor; }
            ENDCG
        }
    }
6

The depth used for image effects with the forward rendering path comes from a depth prepass which uses the shadow caster pass(es) in your shader. Whether or not your shader’s forward pass does zwrite or not is actually irrelevant. Furthermore all passes zwrite by default so adding ZWrite On to a shader that otherwise does not define that behavior will have no effect.

7

ok - solution?

8

You effectively need two shader caster passes, one for the base mesh and a second custom one for the outline that disables itself when casting shadows.

If you add a shadowcaster pass to the shader that uses the same surface offsetting code that the outline shader uses you’ll get it to show up in the depth, but disabling it from rendering in the shadows is a bit of a pain.

Unfortunately there’s no 100% way to test in the shadowcaster shader if it’s currently being rendered for the camera depth texture or a shadow map. Unity uses unity_LightShadowBias.z > 0.0 in their own code, but this humorously only works on directional lights as it’s zero for camera depth, spot lights, and point lights. Using unity_LightShadowBias.x > 0.0 is a little more reliable as long as you have at least a bias set on all of your lights, which you should. The unity_LightShadowBias.x value is the light’s bias, and unity_LightShadowBias.z is supposed to be the normal bias though as mentioned it appears to be disabled at least in 5.4.

Now if you add a custom shadowcaster pass to your shader the main model will no longer cast shadows or show up in the depth! The reason for this is the shadow caster pass for the main body was coming from the Fallback, but it sees a shadowcaster pass already exists in the shader and only the first one gets used. So now you can only choose to get the depth from the outline or depth and shadows from the body but not both!

So the answer is there is no solution, at least not with basic shaders alone. So now what?

You can split up the outline and main body into two separate objects and apply separate shaders on each. Has the benefit of you can just disable shadow casting on the outline mesh object. This is kind of clunky though.

You can use the deferred rendering path and write custom deferred toon shaders as in the deferred rendering path the depth really does come from if the shader has ZWrite on or off. But that’s not good for mobile or low end machines and you’re paying the heavy cost of the deferred pipeline.

You can use geometry shaders to double the geometry instead of doing it in two passes. This isn’t something I’m going to spend the time to implement.

You can double the geometry yourself in the model rather than rely on the shader. This is easily the best option, though it’d take some additional shader work to get the outlines to work the same it would still be possible. A bonus is now your objects use half the draw calls!

We can complain at Unity to let more than one shadow caster pass in a shader get used. Since this has been an issue for 5 years I doubt that’ll get changed any time soon.