I have a model which uses a custom alphablend shader to render decals on top of solid geometry. However, when I lightmap the geometry, the decal casts ambient occlusion onto the model beneath it.
Is there a way of preventing the decal from casting AO? For example, is there a shader constant that I can enable? I tried using the _Transmission attribute and including “Transparent” in the shader path/name, and while this uses the transparency to mask the AO, it doesn’t remove it completely.
AO is rendered as a post process from the depth buffer. Decals usually render with depth bias (so in the depth buffer they appear closer to the camera), causing your problem. What you need to do is to use the bias when checking for depth, but not write the decal depth. You can change these properties in the shader that renders your decals.
Never mind. I found a solution by altering the placement of the decal.
Unity’s lightmapper lets you place a decal between 0m and 0.00029m from a surface before it generates AO. Adding an offset to the decal shader prevents any Z fighting issues from such a close placement.