Custom Lighting Function only works in transparent render queue?

Unity Engine
1

Hello,

I’m working on a shader for a procedurally generated landscape, that comebines a basic surface shader with vertex colors with a secondary flat colored surface shader with a custom lighting model. Here is my shader code so far:

Shader "Custom/VertexColorLandscapeShader" {
    Properties {
        _Smoothness("Smoothness", Range(0,1)) = 0.5
        _Metallic("Metallic", Range(0,1)) = 0.0
        _Pov("Point of View", Vector) = (0,0,0,0)
        _Color("Color Back", Color) = (0,1,0,1)
        _CutOffDistance("Cut off at distance", Float) = 32
        _FlattenDistance("Flatten at distance", Float) = 31
    }
    SubShader {
        Tags { "RenderType"="Opaque" }
        Cull Back
    
        CGPROGRAM
        #pragma surface surf Standard fullforwardshadows
        #pragma target 3.0

        struct Input {
            float3 worldPos;
            float4 color : COLOR;
        };

        fixed4 _Color;
        float4 _Pov;
        float _CutOffDistance;
        float _FlattenDistance;

        void surf (Input IN, inout SurfaceOutputStandard o) {
            o.Albedo = IN.color;
            if(distance(float3(IN.worldPos.x, 0, IN.worldPos.z), float3(_Pov.x, 0, _Pov.z)) > _FlattenDistance-.5) discard;
        }
        ENDCG

        Cull Back

        CGPROGRAM
        #pragma surface surf _Uniform
        #pragma vertex vert
        #pragma target 3.0
        #include "UnityPBSLighting.cginc"

        struct Input {
            float3 worldPos;
        };

        fixed4 _Color;
        float4 _Pov;
        float _CutOffDistance;
        float _FlattenDistance;

        inline half4 Lighting_Uniform(SurfaceOutputStandard s, half3 viewDir, UnityGI gi) {
            half4 c;
            c.rgb = s.Albedo * (_LightColor0.rgb + UNITY_LIGHTMODEL_AMBIENT.rgb);
            c.a = 1;
            return c;
        }

        //New Standard Lighting model requires a GI function
        inline void Lighting_Uniform_GI(
            SurfaceOutputStandard s,
            UnityGIInput data,
            inout UnityGI gi)
        {
            gi = UnityGlobalIllumination (data, s.Occlusion, s.Smoothness, s.Normal);
        }

        void vert(inout appdata_full v, out Input o) {
            UNITY_INITIALIZE_OUTPUT(Input,o);
            float3 worldPos = mul(unity_ObjectToWorld, v.vertex);
            if(distance(float3(worldPos.x, 0, worldPos.z), float3(_Pov.x, 0, _Pov.z)) > _FlattenDistance) v.vertex.y = -0.5;
        }

        void surf (Input IN, inout SurfaceOutputStandard o) {
            o.Albedo = _Color;
            if(distance(float3(IN.worldPos.x, 0, IN.worldPos.z), float3(_Pov.x, 0, _Pov.z)) < _FlattenDistance-.5) discard;
            if(distance(float3(IN.worldPos.x, 0, IN.worldPos.z), float3(_Pov.x, 0, _Pov.z)) > _CutOffDistance+.5) discard;
            if(IN.worldPos.y < -0.2) discard;
        }
        ENDCG
    
    }
    FallBack "Diffuse"
}

The problem is: it only works on the transparent render queue. when set to anything else (geometry or alpha test) the parts with the custom lighting are not drawn at all. There are no errors or warnings, they are just invisible (the solid grey parts in the first image):

1524×790 72 KB

set to geometry:

1524×790 74.1 KB

I’ve got no idea why this is happening, and I’d be happy about any hints in the right direction.

Thank you.

2

(Edited the question and thread title since my problem evolved, and my first attempt was badly described anyways)

3

Are you using the deferred rendering path?

Custom lighting functions don’t work in deferred, so it’ll fall back to forward rendering when it’s used normally. However in this case because there are two Surface shaders the first one is generating a deferred pass. If the render path is set to deferred and a shader had a deferred pass, it ignores all forward passes.

1 Like
4

You are right, bgolus, switching to forward path solved it. Thank you.

Sadly, SSAO and some other image effects don’t work correctly in forward path with cutout shaders: Unity Issue Tracker - SSAO in Forward rendering does not use Cutout from Shader

SSAO is visible on transparent pixels:

1656×680 52.9 KB

The documentation is very vague on custom lighting in deferred path, but I found this: Unity - Manual: Custom lighting models in Surface Shaders
I have no idea how to get the desired effect out of this. What I got so far is:

    inline half4 Lighting_Uniform_Deferred (SurfaceOutputStandard s, UnityGI gi, out half4 outDiffuseOcclusion, out half4 outSpecSmoothness, out half4 outNormal) {
        outDiffuseOcclusion = half4(_Color.rgb, 1);
        outSpecSmoothness = half4(0,0,0,0);
        outNormal = half4(1,1,1,0);

        return outDiffuseOcclusion;
    }

It works, but is far from what I tried to achieve:

1656×680 59.8 KB

Maybe my best chance is to wait for unity to resolve the SSAO issue and disable it for now. Or is there a way to get a flat color effect with a deferred custom lighting function?

5

The bug that person is having is you need to set your shader to use Tags { "RenderType"="TransparentCutout" } to let Unity know it needs to be treated as a cutout for the depth normals texture, or if you’re doing vertex modifiers you need to add a custom RenderType and update the Internal-DepthNormalsTexture.shader and override it in the graphics settings.

Yeah, I just skip surface shaders entirely when I’m doing custom deferred stuff. However try setting the outDiffuseOcclusion to half4(0,0,0,1) and returning the same half4(_Color.rgb, 1.0) you already are. You don’t want to set the diffuse color since you don’t want it to be affected by lighting. The value returned by that function should be the ambient lighting * diffuse color + emission normally. In this case since you just want a solid color, just return the solid color.

6

Thank you for your help so far. I tried some of your suggestions:

Well, the second part of that went right over my head. I tried setting RenderType to TransparentCutout, but this does not do anything in regards to SSAO (or any other image effects BTW).

That works - except it misses with SSAO as well. Still this seems to be the most promising direction to take, so I will experiment with it some more. Thanks again bgolus!