[Unity 5.6.0f3] empty stencil buffer OnRenderImage

Unity Engine
1

Hi all

I have very simple stencil based image effect that perfectly works in Unity 5.5.2 all it does is “bit mask image” output.

The problem is that this effect in Unity 5.6.0f3 gives solid black screen and it looks like stencil buffer is empty.

Its very critical for me to fix this problem using 5.6.0f3 version or to find other solution.
I will be very grateful to any answer.

Thanks

Image effect screenshot from unity 5.5.2

Effect code example

    private void OnRenderImage (RenderTexture source, RenderTexture destination) {
        Graphics.Blit (source, destination, effectMaterial);
    }

Objects shader example

     SubShader {
         Pass {
             Stencil {
                 Ref 128
                 Pass Replace
             }

             CGPROGRAM
             #pragma vertex vert
             #pragma fragment frag

             #include "UnityCG.cginc"

             float4 vert (float4 vertex : POSITION) : SV_POSITION{
                 return UnityObjectToClipPos(vertex);
             }

             fixed4 frag () : SV_Target {   
                 return fixed4(0.5, 0.5, 0.5, 1);
             }
             ENDCG
         }
     }

Image effect shader example

    SubShader {
         Pass {
             Cull Off ZWrite Off ZTest Always
             Stencil {
                 Ref 128
                 Comp NotEqual
             }

             CGPROGRAM
             #pragma vertex vert
             #pragma fragment frag

             #include "UnityCG.cginc"

             float4 vert (float4 vertex : POSITION) : SV_POSITION{
                 return UnityObjectToClipPos(vertex);
             }

             fixed4 frag () : SV_Target {   
                 return fixed4(0, 0, 0, 1);
             }
             ENDCG
         }

         Pass {
         Cull Off ZWrite Off ZTest Always
             Stencil {
                 Ref 128
                 Comp Equal
             }

             CGPROGRAM
             #pragma vertex vert
             #pragma fragment frag

             #include "UnityCG.cginc"

             float4 vert (float4 vertex : POSITION) : SV_POSITION{
                 return UnityObjectToClipPos(vertex);
             }

             fixed4 frag () : SV_Target {   
                 return fixed4(1, 1, 1, 1);
             }
             ENDCG
         }
     }
2

First of all, I would not recommend just setting the reference value to 128. Instead you should keep the reference value to the default value of 255 and set the ReadMask and WriteMask to 128. This will give you an actual check on a single bit.

In the current case you are writing and checking 10000000 binary. In the latter case you are writing and checking 1xxxxxxx, where the x’s are ignored.

Unity probably also uses the stencil buffer here and there and you will need to try and coexist with that. So now that you are actually only looking at a single bit instead of all eight, you can try each bit to see whether that works. (1, 2, 4, 8, 16, 32, 64 and 128.)

3

Hi jvo3dc, thanks for your reply and recommendations.
I’ve already checked each bit and result is the same.

Today I was able to achieve same result with command buffer in unity 5.6.0f3.
But currently I’m facing new problem cause this effect is used to detect if object is visible through opaque enviroment by reading pixels to a texture and checking their color. After pixels are read I need to return to image without any effects and I have no idea on how to achieve this kind of functionality using command buffer (I’m new to it).

So currently I need to understand if its a bug in unity that will be fixed in next versions or they’ve changed the API and added more steps for this kind of functionality in OnRenderImage or maybe completely removed it.

4

It could well be a “bug”. On the other hand, it’s not really clear which parts of the stencil buffer are used when by Unity itself and which are free for us to use.

If you can get in between before it is cleared using a CommandBuffer you would have a good workaround, but they indeed require some getting used to.

5

Solved problem with CommandBuffer.

    public Material effectMaterial;
    private CommandBuffer commandBuffer;

    private Texture2D m_Texture;
    private Texture2D texture {
        get {
            if (m_Texture == null) {
                m_Texture = new Texture2D(16, 16);
            }

            return m_Texture;
        }
    }

    private Camera m_Camera;
    public Camera camera {
        get {
            if (m_Camera == null) {
                m_Camera = GetComponent<Camera> ();
            }

            return m_Camera;
        }
    }
       
    private void Start () {
        if (commandBuffer == null)
        {
            commandBuffer = new CommandBuffer();
            commandBuffer.name = "commandBuffer";

            int stencilTextureID = Shader.PropertyToID("_StencilTexture");
            commandBuffer.GetTemporaryRT(stencilTextureID, -1, -1, 24);

            commandBuffer.Blit(BuiltinRenderTextureType.None, stencilTextureID, effectMaterial);

            commandBuffer.SetGlobalTexture("_StencilTexture", stencilTextureID);
            camera.AddCommandBuffer(CameraEvent.AfterForwardAlpha, commandBuffer);
        }
    }

    private void OnRenderImage (RenderTexture source, RenderTexture destination) {
        RenderTexture stencilRT = Shader.GetGlobalTexture ("_StencilTexture") as RenderTexture;
        RenderTexture activeRT = RenderTexture.active;
        RenderTexture.active = stencilRT;

        texture.ReadPixels (new Rect (Screen.width / 2, Screen.height / 2, texture.width, texture.height), 0, 0, false);
        //Debug.Log (texture.GetPixel(0, 0));

        RenderTexture.active = activeRT;

        Graphics.Blit (source, destination);
    }
6

Found something interesting. If MSAA is turned ON on the rendering camera you’ll get black texture. From my point of view this functionality isn’t connected with MSAA. So its very odd. And now I need to solve this some how

7

Well, MSAA does also need to be applied to the depth/stencil buffer besides the color target. So at 4x MSAA, the depth/stencil buffer should also be able to store 4 samples per pixel.

8

MSAA Solution

    private void OnEnable () {
        if (commandBuffer == null)
        {
            commandBuffer = new CommandBuffer();
            commandBuffer.name = "commandBuffer";

            int cachedScreenImageID = Shader.PropertyToID("_Temp");
            commandBuffer.GetTemporaryRT(cachedScreenImageID, -1, -1, 0);

            commandBuffer.Blit(BuiltinRenderTextureType.CameraTarget, cachedScreenImageID);
            commandBuffer.Blit(cachedScreenImageID, BuiltinRenderTextureType.CameraTarget, effectMaterial);
        
            commandBuffer.SetGlobalTexture("_CachedScreenImage", cachedScreenImageID);
            camera.AddCommandBuffer(CameraEvent.AfterForwardAlpha, commandBuffer);
        }
    }

    private void OnRenderImage(RenderTexture source, RenderTexture destination) {
        RenderTexture cachedScreenRT = Shader.GetGlobalTexture("_CachedScreenImage") as RenderTexture;

        texture.ReadPixels(new Rect(Screen.width / 2, Screen.height / 2, texture.width, texture.height), 0, 0, false);
        Debug.Log(texture.GetPixel(0, 0));

        Graphics.Blit(cachedScreenRT, destination);
    }
1 Like
9 
 private void OnEnable()
    {
        if (commandBuffer == null)
        {
            commandBuffer = new CommandBuffer();
            commandBuffer.name = "commandBuffer";
            commandBuffer.Blit(BuiltinRenderTextureType.CameraTarget, BuiltinRenderTextureType.CameraTarget, postStencilRed);
            camera.AddCommandBuffer(CameraEvent.AfterForwardAlpha, commandBuffer);
        }
    }
2 Likes