Help needed with Surface Shader custom lighting to apply custom shadow

Unity Engine
, ,
1

I’ve been working on a water shader with refraction and foam. You can see a video on Mastodon here:

I’m happy with lots of aspect of it, but I need the foam to receive shadows which is tricky with an alpha blended surface shader since they don’t support it out of the box.

I found the shadow workaround mentioned [here]( No shadows visible on Transparency Shaders?!! page-3#post-4914971) and got as far as getting a shadow multiplier value I can multiply onto my Albedo value. But that’s incorrect, as it makes the shadows also appear on surfaces facing away from the sun, which thus get “double shadow”.

So I need to modify the function that applies lighting from the scene’s directional light, and that’s why I looked into custom lighting functions for Surface shader. And then I got confused.

I started out with the lighting examples on the Surface Shader Lighting Examples page, which specify a function signature like this:

half4 LightingSimpleLambert (SurfaceOutput s, half3 lightDir, half atten)

I multiplied the shadow multiplier onto the light from the dot product (as well as figured out I have to call saturate() on the dot product to clamp it to the 0-1 range, and also have to multiply the rgb with the alpha value since it’s premultiplied alpha - the examples don’t show this for some reason). But it worked, my shadow now only removes light; it doesn’t make the dark side of an object even darker.

However, now ambient lighting behaves wrong. The entire surface is lit up by ambient lighting, also the areas where alpha is zero. So I need to apply the alpha to the rgb contribution from ambient light, but I don’t know where and how.

I look at a lot of forums and blog posts about Surface shader custom lighting functions, but I only get more confused. I try to look at the official page Custom lighting models in Surface Shaders to hopefully get an overview, but FFS, this page says to use function signatures that are completely different than the ones the official examples use, like these two:

half4 Lighting<Name> (SurfaceOutput s, half3 viewDir, UnityGI gi)
and
half4 Lighting<Name>_GI (SurfaceOutput s, UnityGIInput data, inout UnityGI gi)

These don’t have a parameter for light direction, and don’t show how to actually calculate lighting, so I have no idea how to do that.

I try to use different combinations of methods I see from the doc page, official examples, and from various forum and blog posts. But since I don’t know which signatures work together, or why there’s different information about what they’re supposed to be, I find it very perplexing.

For example, this blog post “My take on shaders: What’s the deal with surface shaders?” lists a GI function that returns void instead of a color. So how is the output color manipulated?

inline void LightingStandardOverride_GI (SurfaceOutputOverride s, UnityGIInput data, inout UnityGI gi)

I also saw this method from Unity’s Lighting.inc include file:

inline fixed4 LightingLambert (SurfaceOutput s, UnityGI gi)
{
    fixed4 c;
    c = UnityLambertLight (s, gi.light);

    #ifdef UNITY_LIGHT_FUNCTION_APPLY_INDIRECT
        c.rgb += s.Albedo * gi.indirect.diffuse;
    #endif

    return c;
}

Is this called once per light or once in total? More confusingly, I tried to use this method, but the part related to gi.indirect.diffuse seemed to have an effect on both indirect light AND emission at the same time. I might be going crazy here though; I couldn’t make heads or tails of it.

How am I supposed to understand how to use which functions for what when every page has different signatures both in input parameters and output? And sometimes, I can use only one lighting function, and the shader compiles fine, but other times if I have one, Unity says I must also have the other (GI). I can’t figure out what the patterns are at all.

What I need is essentially this:

  • Calculate lighting per light. I need to be able to modify the result of the dot product. I would also need to apply highlights, so view direction is needed. I’d want parity of how Unity does it normally for surface shaders.

  • Get ambient lighting, whichever type the scene’s lighting settings is set to. Built-in would be fine, but ideally I’d like to be able to modify the result too.

  • Get reflections, handled the same way as standard Surface shader, but with the ability to modify it before it’s applied.

Which functions do I need to use to handle all of these and which functionality go in which functions? Since the shader is alpha-blended I assume it would always be forward-rendered and deferred rendering support is not relevant.

I hope someone can help me make sense of it all.

2

I can see it’s not just me, there’s frequently people who have gotten confused over why the official examples don’t match the function signatures that the official docs say to use. For example here and here.

Going back to earlier versions of the documentation, I can see the official page matches the examples back in early Unity 5. However, with the old style signatures that take a light direction, it’s not clear how to handle ambient lighting, which seems to be added automatically. And with the new signatures that take GI parameters, I haven’t seen examples yet of how to actually perform calculations per light at all, let alone have control over ambient light too.

I’ve now tried to copy in the functions LightingStandard and LightingStandard_GI from UnityPBSLighting.cginc and renamed them in my shader to LightingWater and LightingWater_GI, as well as telling the shader #pragma surface surf Water. Even though my methods are identical, alpha doesn’t work correctly: now have to multiply the output color with the output alpha inside the LightingWater function even though this is not done in the builtin LightingStandard function.

Edit: That turned out to be related to an interference with finalcolor (see post below) - when not using finalcolor, this is not neccesary, but I do need that finalcolor though.

So clearly, even though telling the #pragma to just use one pair of function instead of another, it doesn’t only do that; it performs other changes behind the scenes too which I have no idea how to counteract and correctly account for to get back to the same results that the builtin Standard lighting has.

3

Well, upon more investigation, the discrepency between the built-in lighting functions and the copy-pasted ones seem to relate to the finalcolor function being treated in a different way.

I have this finalcolor function in my shader in order to handle water transparency correctly, as described in this tutorial by catlikecoding:

        void ResetAlpha (Input IN, SurfaceOutputWater o, inout fixed4 color) {
            color.a = 1;
        }

If I comment out color.a = 1; then the copy-pasted lighting functions behave the same as the builtin ones (they both produce wrong results for the purposes of this shader, but identical wrong results).

However, when the finalcolor function is used (and sets the alpha to 1) then the copy-pasted lighting functions work in a different way than the builtin ones. It actually seems like setting the alpha to 1 in finalcolor has no effect when the custom lighting functions are used. The finalcolor method is used though - if I set the color to red, the shader is all red. But the value of the finalcolor alpha does not seem to have any effect.

Something related to when the finalcolor method is called or how it is used must be switched behind the scenes when telling Unity to use a different lighting function? I still don’t understand why, or how to fix it.

4

Well, the plot thickens. If I use Unity’s own SurfaceOutputStandard struct instead of my copy-pasted identical SurfaceOutputWater struct, then I can get identical results, including with use of the finalcolor function. So I’ve now managed to get one-to-one parity with the builtin lighting functions using my own copy-pasted ones.

Unfortunately I need to pass some custom data to the custom lighting functions and I don’t know how to do that if I can’t switch SurfaceOutputStandard to a different struct with additional information.

5

Ok, problem finally solved. It turns out the reason was that when switching from using SurfaceOutputStandard to a different struct (whether identical or not), Unity decides to change the alpha blending mode, so I had to manually set it to alpha:premul.

Not only that, but to ACTUALLY have it use premultiplied alpha for real, I also had to define this variable in the shader:
#define _ALPHAPREMULTIPLY_ON 1

AND to have that actually have any effect, I had to include the Lighting.cginc include file right afterwards.

I found these solutions via this thread:

Feck me this is a mess of a system, but at least I got it to work eventually.