Custom Sprite Lit Shader - Fog of War effect impossible with normal maps?

[mephystpixel]

Hello everyone. I’ve run into an issue with a Custom Sprite Lit shader that i have no idea how to solve, and I’m posting out of desperation.

My goal is to use the 2D Lighting System with one lightmap for regular lights, and a second lightmap for a “Fog of War” effect, where enemy sprites outside of these light sources are invisible, and enemies inside are visible.
The reason i’m not just using a sprite mask for this is that I want enemies to also be hidden by the shadows projected by Shadowcasters, so i need to use the light simulation. In summary:

Lightmap 1: the base color of sprites is multiplied by the color of this lightmap
Lightmap 2: the alpha of sprites is lowered the less intense this lightmap is

I have achieved the desired effect by writing a Custom Sprite Lit Shader Graph, but there’s an issue with normal maps that i can’t wrap my head around. Here’s a screenshot of the shader:

I hope everything is readable.

And here’s a screenshot of the shader in action:

Every sprite in this scene is using a material with the above shader. The 3 blocks in the middle have “UsesFogOfWar” set to true.
The 2 regular lights (blue and orange) are correctly illuminating the floor tiles and the wall on the right (2 Tilemaps), and interacting with normal maps.
There’s also an invisible Fog of War light that is exposing the 2 blocks inside its area.
Unfortunately, it appears that the normal maps of the floor are bleeding trough the blocks, and also the normal map of the blocks is not being hidden outside the Fog of War light.

Bizarrely, if i set “UsesFogOfWar” to false on the material of the 3 blocks, the normal map bleeding stops:

Of course, the Fog of War effect stops as well, and all 3 blocks become fully visible.

So the issues are:

1. Why is the normal map of the blocks visible OUTSIDE the Fog of War light?
   I am hard setting the alpha to 0, so i don’t see how any colors should be visible at all, unless Unity is doing some secret normal map calculations after my shader executes, and adding the extra color on top at the end.
   I can’t even manipulate the alpha of the normal map depending on the Fog of War lightmap, because the Normal block in the Fragment only takes RGB, it has no alpha. There’s no way to fade the normal map out, or change its “strength”. The “Normal Strength” node would only make it more neutral, which would still leave it visible when lights are directly above.
2. Why are the normal maps of the floor showing up ABOVE the blocks? and ONLY when UsesFogOfWar is true? It feels like i’m doing something that is screwing with Unity’s ability to surmise the correct draw order.

I’ve played around with Alpha Clipping, which in theory should help me hide pixels outside the FOW light and solve problem 1, but it also appears to completely disable all normal maps on the object, for some reason.

I’m open to any and all suggestions, and I apologize if I missed something simple. I’m a beginner and I have no idea what I’m doing.
I’m using Unity 6.3.0f1.

Thanks

[mephystpixel]

I made a step forward today, after many many hours of throwing stuff at the wall to see what sticks.

I found a setting of Renderer 2D that half-fixes the bleeding normals problem, but only if i set it to “Floor”, which is the sorting layer of the floor tiles. If i set it to anything else, i get the same outcome as the post above.

The documentation page that mentions that setting is borderline incomprehensible to me, unfortunately.

This is the current situation with that setting:

As you can see:

1. The normals of the floor tiles no longer bleed through the blocks above
2. The normals of the blocks are no longer visible outside of the Fog of War light

However:

1. The normals of each block still bleed through to other blocks that intersect them.
2. The overall brightness of the blocks is far darker than it should. I have no idea why this is. If multiply the intensity of regular lights for just the blocks by 4 it almost fixes the problem, but I’m sure there’s some missing calculation i should be doing in my shader somewhere to fix this properly.

[venediklee]

@mephystpixel

Custom Sprite Lit

Sprite custom lit shaders are used for creating a custom lighting model, not for changing the alpha etc. For example: if you want the normals of an object to react differently to lights based on some parameter or texture( you can do this per blend mode – there are 4 available and customizable in the 2D renderer iirc;) you’d use sprite custom lit. Other use cases/examples can be found in the docs and forum etc.

So basically I dont think you need to use sprite custom lit, just use sprite lit and the lighting calculations will be done automatically for you. If the object needs to be transparent you can do so by changing the alpha in a regular sprite lit shader like you have done before. You’ll probably want to revert the camera sorting layer too

Also if you are wondering why some things are rendering the way they are you can check the frame debugger

Cheers

[mephystpixel]

Thank you very much for responding.

A custom lighting model is exactly what i need, unfortunately. The effect i want to achieve is the field of view from Darkwood, where enemies disappear when you’re looking away.
I also want to render the environment in grayscale if it’s outside the field of view.
I don’t think any of these effects are possible without using Custom Sprite Lit.

I managed to make all of it work except for the normal maps problem. The shader i posted above is my minimal reproduction of the issue. Here’s a screenshot of the complete shader with the same problem (the 4 blocks are placeholder enemies):

Thank you also for suggesting the Frame Debugger, i wasn’t familiar with it and it’s full of interesting information.

It seems that the Normal2D Pass is done immediately, first for the tilemap:

Then for the “SRP Batch”:

And we can see that the alpha of my shader is completely ignored here, and there’s also some weird transparency going on, which causes the normals of the floor to be visible through the blocks.

Afterwards, everything else is rendered correctly, including my custom sprite alpha. (result is the first image of this post)

I honestly can’t find any useful information in the details of the pass. The shader at the bottom is my Custom Sprite Lit shader.

I’m either missing something or this is some kind of bug.

I’d also like to add that the solution i mentioned above of changing Camera Sorting Layer Texture options is terrible, as in the Light Batching Debugger i can see that it breaks light batching, doubling all light calculations and removing ~80fps on my machine.

[venediklee]

@mephystpixel

And we can see that the alpha of my shader is completely ignored here

It must be ‘ignored’ because normals pass doesnt know how the normals will be used in the next passes + it allows the user to do custom stuff based on the rendered normals

@mephystpixel

some weird transparency going on

(I can see the blocks are transparent) I dont exactly know why this is but it could be just the normal calculation multiplies the normal values of the fragment with the alpha value; which makes sense because it wouldnt be correct otherwise if a transparent lit object overlapped another object.

@mephystpixel

breaks light batching, doubling all light calculations and removing ~80fps on my machine.

thanks for the info