[Solved] Outline custom pass example, how to exclude occluded objects?

I’ve been trying to recreate my former highlight fx system in the HDRP, and used outline custom example from the manual.

However, outline are drawn on top of everything. Is there a way to draw it only if the object is visible?

@antoinel_unity I’ve tried this example HDRP-Custom-Passes/Assets/CustomPasses/Selection/Shaders/02_Selection_Fullscreen.shader at master · alelievr/HDRP-Custom-Passes · GitHub and I can’t make inner / outer color to hide correctly behind objects;

It is still visible behind other objects even with _BehindFactor 0;
Is depth actually broken?

This is what I’ve tried:

float4 FullScreenPass(Varyings varyings) : SV_Target
    {
        float depth = LoadCameraDepth(varyings.positionCS.xy);
        PositionInputs posInput = GetPositionInput(varyings.positionCS.xy, _ScreenSize.zw, depth, UNITY_MATRIX_I_VP, UNITY_MATRIX_V);
        float3 viewDirection = GetWorldSpaceNormalizeViewDir(posInput.positionWS);
     
        float d = LoadCustomDepth(posInput.positionSS);
        float db = LoadCameraDepth(posInput.positionSS);
     
        float4 c = LoadCustomColor(posInput.positionSS);

        float obj = c.a;
     
        uint offset = 5;
     
        int sampleCount = min( 2 * pow(2, _SamplePrecision ), MAXSAMPLES ) ;
     
        float4 outline = float4(0,0,0,0);
     
        float2 uvOffsetPerPixel = 1.0/_ScreenSize .xy;
     
        for (uint i=0 ; i<sampleCount ; ++i )
        {
            outline =  max( SampleCustomColor( posInput.positionNDC + uvOffsetPerPixel * _OutlineWidth * offsets[i] ), outline );
        }

        float4 o = float4(0,0,0,0);
     
        float alphaFactor = (db > d) ? _BehindFactor : 1;
     
        float4 innerColor = SAMPLE_TEXTURE2D(_Texture, s_trilinear_repeat_sampler, posInput.positionSS / _TextureSize) * _InnerColor;
        innerColor.a *= alphaFactor;
     
        float4 outerColor = _OuterColor * float4(outline.rgb, 1);
        outerColor.a *= alphaFactor;
        outline.a *= alphaFactor;
     
        o = lerp(o, outerColor, outline.a);
        o = lerp(o, innerColor * float4(c.rgb, 1), obj);
     
        return o;
    }

This is my whole setup:

Draw renderers pass:

using UnityEngine;
using UnityEngine.Experimental.Rendering;
using UnityEngine.Rendering;
using UnityEngine.Rendering.HighDefinition;

namespace FX.Highlight {
   public class HighlightPassDrawRenderers : CustomPass {
      //Filter settings
      public LayerMask LayerMask = 1; // Layer mask Default enabled

      [SerializeField]
      private float _maxDistance = 15f;
  
      [SerializeField]
      private float _lerpMaxDistance = 15f;

      [SerializeField]
      private Shader _shader;

      #region [Fields]

      private static ShaderTagId[] _forwardShaderTags;
  
      private static readonly int MaxDist = Shader.PropertyToID("_MaxDistance");
      private static readonly int LerpMaxDist = Shader.PropertyToID("_LerpMaxDistance");

      // Cache the shaderTagIds so we don't allocate a new array each frame
      private ShaderTagId[] _cachedShaderTagIDs;

      private Material _material;
      private int _pass;

      #endregion

      protected override void Setup(ScriptableRenderContext renderContext, CommandBuffer cmd) {
         _shader = Shader.Find("HDRP/PostFX/HighlightPassDrawRenderers");
         _material = CoreUtils.CreateEngineMaterial(_shader);

         _pass = _material.FindPass("FirstPass");
     
         _forwardShaderTags = new[]
                              {
                                 new ShaderTagId("Forward"), // HD Lit shader
                                 new ShaderTagId("ForwardOnly"), // HD Unlit shader
                                 new ShaderTagId("SRPDefaultUnlit"), // Cross SRP Unlit shader
                                 new ShaderTagId(""), // Add an empty slot for the override material
                              };
      }

      protected override void AggregateCullingParameters(ref ScriptableCullingParameters cullingParameters,
                                                         HDCamera hdCamera) {
         cullingParameters.cullingMask |= (uint) (int) LayerMask;
      }

      /// <summary>
      /// Execute the DrawRenderers with parameters setup from the editor
      /// </summary>
      protected override void Execute(ScriptableRenderContext renderContext,
                                      CommandBuffer cmd,
                                      HDCamera hdCamera,
                                      CullingResults cullingResult) {
         SetRenderTargetAuto(cmd);
     
         var stateBlock = new RenderStateBlock();

         PerObjectData renderConfig = hdCamera.frameSettings.IsEnabled(FrameSettingsField.Shadowmask)
                                         ? HDUtils.k_RendererConfigurationBakedLightingWithShadowMask
                                         : HDUtils.k_RendererConfigurationBakedLighting;

         var result = new RendererListDesc(_forwardShaderTags, cullingResult, hdCamera.camera)
                      {
                         rendererConfiguration = renderConfig,
                         renderQueueRange = RenderQueueRange.all,
                         sortingCriteria = SortingCriteria.BackToFront,
                         excludeObjectMotionVectors = false,
                         overrideMaterial = _material,
                         overrideMaterialPassIndex = _pass,
                         stateBlock = stateBlock,
                         layerMask = LayerMask,
                      };

         _material.SetFloat(MaxDist, _maxDistance);
         _material.SetFloat(LerpMaxDist, _lerpMaxDistance);
     
         HDUtils.DrawRendererList(renderContext, cmd, RendererList.Create(result));
      }

      protected override void Cleanup() {
         base.Cleanup();
     
         CoreUtils.Destroy(_material);
      }
   }
}

Draw renderers shader:

Shader "HDRP/PostFX/HighlightPassDrawRenderers"
{
    Properties
    {
        _MaxDistance("Max Distance", float) = 15
        _LerpMaxDistance("Lerp Max Distance", float) = 20
    }

    HLSLINCLUDE

    #pragma target 4.5
    #pragma only_renderers d3d11 ps4 xboxone vulkan metal switch

    // #pragma enable_d3d11_debug_symbols

    //enable GPU instancing support
    #pragma multi_compile_instancing
    #pragma instancing_options renderinglayer
 
    ENDHLSL

    SubShader
    {
        Pass
        {
            Name "FirstPass"
            Tags { "LightMode" = "FirstPass" }

            Blend Off
            ZWrite On
            ZTest LEqual

            Cull Back

            HLSLPROGRAM

            // Toggle the alpha test
            #define _ALPHATEST_ON

            // Toggle transparency
            // #define _SURFACE_TYPE_TRANSPARENT

            // Toggle fog on transparent
            #define _ENABLE_FOG_ON_TRANSPARENT
        
            // List all the attributes needed in your shader (will be passed to the vertex shader)
            // you can see the complete list of these attributes in VaryingMesh.hlsl
            #define ATTRIBUTES_NEED_TEXCOORD0
            #define ATTRIBUTES_NEED_NORMAL
            #define ATTRIBUTES_NEED_TANGENT

            // List all the varyings needed in your fragment shader
            #define VARYINGS_NEED_TEXCOORD0
            #define VARYINGS_NEED_TANGENT_TO_WORLD
            #define VARYINGS_NEED_POSITION_WS
        
            #include "Packages/com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/RenderPass/CustomPass/CustomPassRenderers.hlsl"
        
            float _MaxDistance;
            float _LerpMaxDistance;
        
            float invLerp(float A, float B, float T)
            {
                return (T - A) / (B - A);
            }

            void GetSurfaceAndBuiltinData(FragInputs fragInputs, float3 viewDirection, inout PositionInputs posInput, out SurfaceData surfaceData, out BuiltinData builtinData)
            {
                float dist = length(fragInputs.positionRWS);
        
                if (dist > _LerpMaxDistance) {
                    discard;
                }
            
                float percentage = invLerp(_MaxDistance, _LerpMaxDistance, dist);
                percentage = clamp (percentage, 0, 1);
            
                // Write back the data to the output structures
                ZERO_INITIALIZE(BuiltinData, builtinData); // No call to InitBuiltinData as we don't have any lighting
                builtinData.opacity = lerp(1, 0, percentage);
                builtinData.emissiveColor = float3(0, 0, 0);
                surfaceData.color = float3(1,1,1);
            }

            #include "Packages/com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/ShaderPass/ShaderPassForwardUnlit.hlsl"

            #pragma vertex Vert
            #pragma fragment Frag

            ENDHLSL
        }
    }
}

Fullscreen pass:

using UnityEngine;
using UnityEngine.Rendering;
using UnityEngine.Rendering.HighDefinition;

namespace FX.Highlight {
   /// <summary>
   /// FullScreen Custom Pass
   /// </summary>
   [System.Serializable]
   public class HighlightPassFullscreen : CustomPass {
      [SerializeField, Range(1, 3)]
      private float _samplePrecision = 1;

      [SerializeField]
      private float _outlineWidth = 5;

      [SerializeField]
      private Color _innerColor = new Color(1, 1, 0, 0.5f);
  
      [SerializeField]
      private Color _outerColor = new Color(1, 1, 0, 0.5f);

      [SerializeField]
      private Texture _texture = default;
  
      [SerializeField]
      private Vector2 _texturePixelSize = new Vector2(64,64);

      [SerializeField, Range(0, 1)]
      private float _behindFactor = 0.2f;
  
      [SerializeField]
      private Shader _shader;

      #region [Fields]

      private int _fadeValueId;
      private Material _material;

      private int _pass;

      private static readonly int SamplePrecision = Shader.PropertyToID("_SamplePrecision");
      private static readonly int OutlineWidth = Shader.PropertyToID("_OutlineWidth");
      private static readonly int InnerColor = Shader.PropertyToID("_InnerColor");
      private static readonly int OuterColor = Shader.PropertyToID("_OuterColor");
      private static readonly int Texture = Shader.PropertyToID("_Texture");
      private static readonly int TextureSize = Shader.PropertyToID("_TextureSize");
      private static readonly int BehindFactor = Shader.PropertyToID("_BehindFactor");

      #endregion

      protected override void Setup(ScriptableRenderContext renderContext, CommandBuffer cmd) {
         _shader = Shader.Find("HDRP/PostFX/HighlightPassFullscreen");
         _material = CoreUtils.CreateEngineMaterial(_shader);

         _fadeValueId = Shader.PropertyToID("_FadeValue");

         _pass = _material.FindPass("FirstPass");
      }

      /// <summary>
      /// Execute the pass with the fullscreen setup
      /// </summary>
      protected override void Execute(ScriptableRenderContext renderContext,
                                      CommandBuffer cmd,
                                      HDCamera hdCamera,
                                      CullingResults cullingResult) {
         ResolveMSAAColorBuffer(cmd, hdCamera);
         SetRenderTargetAuto(cmd);

         _material.SetFloat(_fadeValueId, fadeValue);
         _material.SetFloat(SamplePrecision, _samplePrecision);
         _material.SetFloat(OutlineWidth, _outlineWidth);
         _material.SetColor(InnerColor, _innerColor);
         _material.SetColor(OuterColor, _outerColor);
         _material.SetTexture(Texture, _texture);
         _material.SetVector(TextureSize, _texturePixelSize);
         _material.SetFloat(BehindFactor, _behindFactor);
     
         CoreUtils.DrawFullScreen(cmd, _material, _pass);
      }
   }
}

Fullscreen shader:

Shader "HDRP/PostFX/HighlightPassFullscreen"
{
    properties
    {
        _SamplePrecision ("Sampling Precision", Range(1,3) ) = 1
        _OutlineWidth ("Outline Width", Float ) = 5
    
        _InnerColor ("Inner Color", Color) = (1, 1, 0, 0.5)
        _OuterColor( "Outer Color", Color ) = (1, 1, 0, 1)
        _Texture ("Texture", 2D ) = "black" {}
        _TextureSize("Texture Pixels Size", Vector) = (64,64,0,0)
    
        _BehindFactor("Behind Factor", Range(0,1)) = 0.2
    }

    HLSLINCLUDE

    #pragma vertex Vert

    #pragma target 4.5
    #pragma only_renderers d3d11 ps4 xboxone vulkan metal switch

    #include "Packages/com.unity.render-pipelines.high-definition/Runtime/RenderPipeline/RenderPass/CustomPass/CustomPassCommon.hlsl"

    // The PositionInputs struct allow you to retrieve a lot of useful information for your fullScreenShader:
    // struct PositionInputs
    // {
    //     float3 positionWS;  // World space position (could be camera-relative)
    //     float2 positionNDC; // Normalized screen coordinates within the viewport    : [0, 1) (with the half-pixel offset)
    //     uint2  positionSS;  // Screen space pixel coordinates                       : [0, NumPixels)
    //     uint2  tileCoord;   // Screen tile coordinates                              : [0, NumTiles)
    //     float  deviceDepth; // Depth from the depth buffer                          : [0, 1] (typically reversed)
    //     float  linearDepth; // View space Z coordinate                              : [Near, Far]
    // };

    // To sample custom buffers, you have access to these functions:
    // But be careful, on most platforms you can't sample to the bound color buffer. It means that you
    // can't use the SampleCustomColor when the pass color buffer is set to custom (and same for camera the buffer).
    // float3 SampleCustomColor(float2 uv);
    // float3 LoadCustomColor(uint2 pixelCoords);
    // float LoadCustomDepth(uint2 pixelCoords);
    // float SampleCustomDepth(float2 uv);

    // There are also a lot of utility function you can use inside Common.hlsl and Color.hlsl,
    // you can check them out in the source code of the core SRP package.
 
    #define v2 1.41421
    #define c45 0.707107
    #define c225 0.9238795
    #define s225 0.3826834
 
    #define MAXSAMPLES 16
    static float2 offsets[MAXSAMPLES] = {
        float2( 1, 0 ),
        float2( -1, 0 ),
        float2( 0, 1 ),
        float2( 0, -1 ),
    
        float2( c45, c45 ),
        float2( c45, -c45 ),
        float2( -c45, c45 ),
        float2( -c45, -c45 ),
    
        float2( c225, s225 ),
        float2( c225, -s225 ),
        float2( -c225, s225 ),
        float2( -c225, -s225 ),
        float2( s225, c225 ),
        float2( s225, -c225 ),
        float2( -s225, c225 ),
        float2( -s225, -c225 )
    };
 
    int _SamplePrecision;
    float _OutlineWidth;
 
    float4 _InnerColor;
    float4 _OuterColor;
 
    Texture2D _Texture;
    float2 _TextureSize;
 
    float _BehindFactor;

    float4 FullScreenPass(Varyings varyings) : SV_Target
    {
        float depth = LoadCameraDepth(varyings.positionCS.xy);
        PositionInputs posInput = GetPositionInput(varyings.positionCS.xy, _ScreenSize.zw, depth, UNITY_MATRIX_I_VP, UNITY_MATRIX_V);
        float3 viewDirection = GetWorldSpaceNormalizeViewDir(posInput.positionWS);
    
        float d = LoadCustomDepth(posInput.positionSS);
        float db = LoadCameraDepth(posInput.positionSS);
    
        float4 c = LoadCustomColor(posInput.positionSS);

        float obj = c.a;
    
        uint offset = 5;
    
        int sampleCount = min( 2 * pow(2, _SamplePrecision ), MAXSAMPLES ) ;
    
        float4 outline = float4(0,0,0,0);
    
        float2 uvOffsetPerPixel = 1.0/_ScreenSize .xy;
    
        for (uint i=0 ; i<sampleCount ; ++i )
        {
            outline =  max( SampleCustomColor( posInput.positionNDC + uvOffsetPerPixel * _OutlineWidth * offsets[i] ), outline );
        }

        float4 o = float4(0,0,0,0);
    
        float alphaFactor = (db > d) ? _BehindFactor : 1;
    
        float4 innerColor = SAMPLE_TEXTURE2D(_Texture, s_trilinear_repeat_sampler, posInput.positionSS / _TextureSize) * _InnerColor;
        innerColor.a *= alphaFactor;
    
        float4 outerColor = _OuterColor * float4(outline.rgb, 1);
        outerColor.a *= alphaFactor;
        outline.a *= alphaFactor;
    
        o = lerp(o, outerColor, outline.a);
        o = lerp(o, innerColor * float4(c.rgb, 1), obj);
    
        return o;
    }

    ENDHLSL

    SubShader
    {
        Pass
        {
            Name "FirstPass"

            ZWrite Off
            ZTest Always
            Blend SrcAlpha OneMinusSrcAlpha
            Cull Off

            HLSLPROGRAM
                #pragma fragment FullScreenPass
            ENDHLSL
        }
    }
    Fallback Off
}

Well, at least I’ve got distance working now.

Depth check (_BehindFactor) doesn’t work correctly if objects are behind anything.
No idea why.

1 Like

Hello, if you want depth test to work with your outline, then it’s probably simpler to render the objects you want to outline with the Z-buffer of the camera so they can be Z-Tested against the scene like this:
5710618--597940--upload_2020-4-14_15-57-12.png

Alternatively, in the fullscreen shader, you could detect edges with the custom depth instead of custom color, it would allow you to perform the Z-Test manually here.

1 Like

Hmmm… I’ve tried these settings. It works weirdly in the scene view (hidden both in front and behind objects). But in game, it still doesn’t work (layer object’s still visible behind objects).

I’ve also tried adding depth comparison to the RenderStateBlock like so (with both writeEnabled false and true), but it changes nothing:

         RenderStateBlock stateBlock = new RenderStateBlock
                                       {
                                          depthState = new DepthState
                                                       {
                                                          compareFunction = CompareFunction.LessEqual,
                                                          writeEnabled = true
                                                       }
                                       };

         PerObjectData renderConfig = hdCamera.frameSettings.IsEnabled(FrameSettingsField.Shadowmask)
                                         ? HDUtils.k_RendererConfigurationBakedLightingWithShadowMask
                                         : HDUtils.k_RendererConfigurationBakedLighting;

         var result = new RendererListDesc(_forwardShaderTags, cullingResult, hdCamera.camera)
                      {
                         rendererConfiguration = renderConfig,
                         renderQueueRange = RenderQueueRange.all,
                         sortingCriteria = SortingCriteria.BackToFront,
                         excludeObjectMotionVectors = false,
                         overrideMaterial = _material,
                         overrideMaterialPassIndex = _pass,
                         stateBlock = stateBlock,
                         layerMask = LayerMask,
                      };

Ha yes, you’re true. You need the whole object in your custom buffer and then compare the depth in the fullscreen pass otherwise it won’t work.

I found an issue in the algorithm of the fullscreen pass: the depth comparison is done with the current value from the custom depth buffer, but because we are doing an outline, all the pixels we care are “outside” of the object we want to outline and this means that the depth value will always be 0 for those pixels.
You need to find the nearest depth value of the object that when doing your search too:

        for (uint i=0 ; i<sampleCount ; ++i )
        {
            outline =  max( SampleCustomColor( posInput.positionNDC + uvOffsetPerPixel * _OutlineWidth * offsets[i] ), outline );
            d = max(SampleCustomDepth(posInput.positionNDC + uvOffsetPerPixel * _OutlineWidth * offsets[i]), d);
        }

I’m getting decent results with this

5715778--598621--Outline.gif

1 Like

Hmmm that somewhat works, but it gives rather inconsistent results when camera is actually moving.

I’ve set draw renderers target depth buffer to the custom and applied change to the shader. And clear flags for the DrawRenderers is set to Color;
5722381--599725--inconsistent.gif

Edit: Alternatively, if you got it working correctly, please upload unitypackage or src somewhere.
I’ve probably messed up my setup somehow. But at this point idk where.

Here is the unity package of the pass: note that I merge the two C# file in one pass because it’s easier to setup :slight_smile:
You can also see in the code a bunch of [Header(…)] commented out, it’s because I found a bug with these decorators attributes and they break the layout of the inspector, should be fixed in the next version of HDRP or so.

5754193–605443–Highligh.unitypackage (24.9 KB)

1 Like

I’ve tried this one, brought it as is to the scene.
And it still renders outline on top of everything, even with behind factor 0.

5762911--606895--upload_2020-4-25_14-22-12.png

Interesting thing, that it seems to work correctly with your scene.
If I bring those outlined objects into it, they aren’t rendered behind other objects.

OMFG. I got it. Its the double camera setup!

I have a separate world space UI camera, which renders on top.
Which probably caused depth to be used from the UI camera instead. Which always empty.

I can’t believe I’ve missed that.

Man, I wish there somewhere target camera being displayed for the pass.
That would’ve saved me a couple of days :slight_smile:

The question is now, how to make sure custom pass uses specific camera?
Edit: Okay, found it.
There’s an override in the Custom Frame Settings in the Camera settings that allows to disable Custom Passes.

Thank you for your support, its really awesome :slight_smile:

3 Likes