How To Visualize Stencil Buffer in Texture

Unity Engine
,
1

Hi guys, I’m currently implementing a shadow volume technique in URP, and I just had a question if something was possible. Essentially, everything about the volume generation and shadows works great; the only problem comes from the lack of blurring in the shadows. Since my shadow extrusion technique is based on triangles that don’t face the light, this causes hard polygonal edges to be extruded (see pic below) for self-shadowing. My idea to solve this problem is to essentially render the stencil buffer as a mask to a render texture, where from there, I could apply a gaussian blur on it and then blit it back to the main camera as a post-processing feature. This would create much more blurry shadows, but would also be a cheap way to fake/emulate soft shadows. Since I’m on URP, I can’t natively use multi-pass in my shaders, so I had to make a render feature to do this (see code below). What I was wondering, is if there is any way for me to set up my feature to be able to pipe in the stencil buffer into a mask texture, or something of the sort. If anyone has any ideas, it would be greatly appreciated, thanks!

9813462--1409730--issue3.PNG
9813462--1409730--issue3.PNG757×315 90.1 KB

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.Rendering;
using UnityEngine.Rendering.Universal;

[System.Serializable]
public class SilhouetteEdgeHighlightSettings
{
    public Material targetMaterial = null; // Assign this in the inspector
}

public class SilhouetteEdgeHighlightFeature : ScriptableRendererFeature
{
    public SilhouetteEdgeHighlightSettings settings = new SilhouetteEdgeHighlightSettings();
   
    class SilhouetteRenderPass : ScriptableRenderPass
    {
        private List<ShaderTagId> m_Tags;
        private Material targetMaterial;
        public SilhouetteRenderPass(List<string> tags, Material material)
        {
            targetMaterial = material;
            m_Tags = new List<ShaderTagId>();
            foreach (string tag in tags)
            {
                m_Tags.Add(new ShaderTagId(tag));
            }
            this.renderPassEvent = RenderPassEvent.AfterRenderingOpaques;
        }
        public override void Execute(ScriptableRenderContext context, ref RenderingData renderingData)
        {
            int layerIndex = 3;
            int layerMask = 1 << layerIndex;
            FilteringSettings filterSettings = new FilteringSettings(RenderQueueRange.opaque, layerMask); // Or any specific range or layer mask
           
            // Iterate over each ShaderTagId you wish to render with the targetMaterial
            foreach (ShaderTagId tag in m_Tags)
            {
                DrawingSettings drawingSettings = CreateDrawingSettings(tag, ref renderingData, SortingCriteria.CommonOpaque);
                context.DrawRenderers(renderingData.cullResults, ref drawingSettings, ref filterSettings);
            }

            context.Submit();
        }
    }

    private SilhouetteRenderPass m_ScriptablePass;
    public List<string> lightModePasses;
    public override void Create()
    {
        m_ScriptablePass = new SilhouetteRenderPass(lightModePasses, settings.targetMaterial);
    }

    public override void AddRenderPasses(ScriptableRenderer renderer, ref RenderingData renderingData)
    {
        if(settings.targetMaterial != null)
        {
            renderer.EnqueuePass(m_ScriptablePass);
        }
    }
}
2

maybe you can find a way to only render the shadows as black etc to a render texture at lower resolution (to make use of bilinear filtering), and then either blit it into your main view or read the render texture in shader with multiple samples to effectively blur it. i’m warning you now though, any form of screenspace blurring will be subject to aliasing/shifting around as the texture changes, so the shadows would look both more and less realistic.

here’s a shadertoy i made demonstrating a way to make bilinear filtering much more finely subdivided with only 3 samples.
3-Tap Smoother Bilinear Filter (shadertoy.com)

here’s a link talking about blurring stencil shadows in screenspace.
Soft Stencil Shadows – The Instruction Limit

note also, you can have overshadowing from between objects as blur in screenspace will bleed over objects.

3

I got it to work! After like months of researching shadow volumes, I actually have a robust shadowing system. Basically, I got the soft shadowing to work by masking the stencil buffer using cmd.DrawMesh. Normal blits don’t preserve stencil values, so I manually did it with the command buffer. Then, I was able to get a mask texture by setting all shadowed areas to white, and all other places colormasked off to make it clear. Also, I was able to do all of this on the main _cameraColorTarget, which allowed my stencils to appear on objects that don’t have the shader enabled for some reason, which is really helpful because it means that I don’t have to enable the shader for all objects in the scene.

Your recommendation on screen-space gaussian blur helped a lot too. I was able to make a shader sampling the shadow texture using depth and normals reconstruction to determine which areas should be blurred more than others, creating realistic shadow blurring. Areas closer to the player appear less blurred, while areas further away are blurrier. This was then blitted back to the main camera, and subsequently blended with the main color to make the stencils be affected by lights, emission, etc. Sorry for writing all of this lol I just thought it might be useful as you told me earlier that you tried to make shadow volumes in the past. I’ll attach an image below too of the stencils exhibiting realistic blur. Thanks for all of your help! :slight_smile:

9821337--1411758--soft.PNG.jpg
9821337--1411761--softshadows.PNG

4

it makes me happy to see you succeed and i’m glad i could help you! :slight_smile:

i’m not sure if i’ll ever go back down the stencil shadow route or not, but if i do decide to i’ll gladly take some inspiration from how you’ve done it.

i must say, it looks pretty dang good in the pictures, so kudos to you on your results :smile: