"Fix" for DirectX Flipping Vertical Screen Coordinates in Image Effects not Working

Unity Engine
1

We have an image effect in our game that is appearing flipped vertically when running on DirectX. This never happened on OpenGL-based platforms. I have read this article and put the block of code that is supposed to fix the issue into the shader, but it is still not working.

// On D3D when AA is used, the main texture and scene depth texture
// will come out in different vertical orientations.
// So flip sampling of the texture when that is the case (main texture
// texel size will have negative Y).

#if UNITY_UV_STARTS_AT_TOP
if (_MainTex_TexelSize.y < 0)
        uv.y = 1-uv.y;

#endif

If I comment out the IF statement testing the texel size, it fixes the issue, but I am worried that if I comment that out, the image might get flipped by the shader during circumstances where it shouldn’t.

I have done some testing and determined that _MainTex_TexelSize is being initialized by Unity (_MainTex_TexelSize.y had a value between 1 / 64 and 1 / 63 when I ran our game. I determined this by an IF-ELSE statement that would make the entire object green if the expression was true, and red if it was false. With this, I could basically do a binary search to figure out the value in the variable…shader debugging, yaaaay!).

I would expect that _MainTex_TexelSize.y should be negative if the image needs to be flipped by the shader; however, it is positive in my shader when the image is clearly upside down and needs to be flipped.

2

Can you post the whole shader ?

3

Sure:

Shader "Custom/CensorFX"
{
    Properties
    {
        _RenderedScene("RenderedScene", 2D) = "white" {}  //Should contain render of scene so far
    }
    SubShader
    {  
        Pass
        {
            ZWrite Off
            ZTest Always

            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag

            #include "UnityCG.cginc"

            sampler2D _RenderedScene;
            float4 _RenderedScene_TexelSize;

            struct VertexInput
            {
                float4 modelPos : POSITION;
            };

            struct PixelInput
            {
                float4 ndcPos : POSITION;
                float4 screenPos : TEXCOORD0; //same as ndcPos.  Have to pass as separate variable to use in pixel shader since ndcPos is "consumed" by the pipeline
                                              //have let GPU interpolate the screen position, then calculate UVs in fragment shader to have mesh appear completely flat
            };

            PixelInput vert(VertexInput vertexInput)
            {
                PixelInput output;
                output.ndcPos = mul(UNITY_MATRIX_MVP, vertexInput.modelPos);
                output.screenPos = output.ndcPos;
                return output;
            }

            float4 frag(PixelInput pixelInput) : COLOR
            {
                //calculate the uv's based on the screen space position so
                //that top left corner in screen space samples from top left
                //of texture, bottom right corner in screen space samples
                //from bottom right of texture, etc...
                //
                //if the _RenderedScene texture is really small and point filtered,
                //this mesh will basically pixelate a certain portion of the screen.
                float2 uv = (pixelInput.screenPos.xy / pixelInput.screenPos.w + 1) / 2;
               
#if UNITY_UV_STARTS_AT_TOP
                if (_RenderedScene_TexelSize.y < 0)  //this is returning false even when the image is upside down!
                {
                    uv.y = 1 - uv.y;
                }
#endif
                return tex2D(_RenderedScene, uv);
            }
            ENDCG
        }
    }
    FallBack "Diffuse"
}

To give you a little more context, this is used for a censorship effect to basically pixelate certain portions of the final image. I will also post the OnRenderImage function I am using in the image effect:

private void OnRenderImage(RenderTexture source, RenderTexture destination)
    {
        if (_censorMeshesToDraw.Count > 0)
        {
            RenderTexture downSampledScreenCap = GetTempRenderTexture();  //scaled down by 8 times and point filtered

            //down sample a smaller version to pass to shader
            //(If I were to save 'downSampledScreenCap' as a png now,
            //it would be right side up)
            Graphics.Blit(source, downSampledScreenCap);

            //pass the full image through the pipeline
            //(If I were to save 'destination' as a png now,
            //it would be right side up).
            Graphics.Blit(source, destination); 

            //pass the scaled down image to the shader, and render the
            //censor meshes with this material.
            _censorMaterial.SetTexture(RENDERERD_SCENE_PROPERTY_NAME, downSampledScreenCap);
            _censorMaterial.SetPass(0);
            foreach (CensorMesh censorMesh in _censorMeshesToDraw)
            {
                Graphics.DrawMeshNow(censorMesh.Mesh, censorMesh.Transform.localToWorldMatrix);
            }

            //If I were to save 'destination' as a png now the original part
            //of the image (the image before censor meshes were drawn) would
            //right side up, but the texture used for the censorship meshes
            //('downSampledScreenCap'), is upside down.

            RenderTexture.ReleaseTemporary(downSampledScreenCap);
        }
        else
        {
            Graphics.Blit(source, destination);  //pass the full image through the pipeline
        }
    }

I think the comments do a pretty good job of explaining how the algorithm works, but let me know if there is anything in the code that you would like me to elaborate on.

Here is some additional information I have found.

  1. My specific problem has nothing to do with antialiasing, like the problem in that article; my problem persists whether or not I have antialiasing on.

  2. The problem only occurs if there is at least one additional image effect after the censorship effect. If there are no additional image effects after this one, everything works fine.

Given observation number 2, I have found a “hack” around the issue. In the OnRenderImage function, it seems that the destination RenderTexture is null if and only if there are no image effects after the current one. So I can test if the destination is not null, and then when I blit the down sampled image, I can flip it with a simple shader on DirectX.

I have tested my “hack” and it seems to work in all cases. But I still don’t feel very satisfied with the solution. There are three things I do not understand:

  1. How is this image is getting flipped in the first place (when I print it out, it is always right side up until the last step)?

  2. Why does the problem only happen when there is an additional image effects after the censorship effect?

  3. Why does Unity’s proposed solution not work in my case?

1 Like
4

So when _censorMeshesToDraw.Count is > 0 it doesn’t work and when it does work it uses blit ?

did you make the censorMesh yourself could the UV’s on V be flipped on model ?

5

The check for _censorMeshesToDraw.Count > 0 is just a safety. If the count is greater than zero, it will do the algorithm like normal, if not, it will effectively do nothing (the code is just blitting source to destination in this case) since there is nothing to censor. However, I have other code to make sure this script is disabled if there is nothing to censor.

The only time it doesn’t work (without the “hack” I mentioned in my previous post), is if I am running on DirectX and if there is another image effect on the camera that has an OnRenderImage function that happens after the censorship effect. If that happens, the censor meshs’ texture will be inverted vertically.

As you can see from the shader I posted, I do not use the UV’s on the meshes. I calculate UV’s in the fragment shader from the ndc space (between -1 and 1 both vertically and horizontally) position of the pixel I am currently shading. So if a pixel on the mesh has a screen space position of (-1, 1) it is in the top left of the screen, and I calculate the UV’s for that pixel to be (0, 1), which is the top left of the texture.

Incidentally, I generally just use Unity’s default sphere for the mesh, since it has a consistent 2D projection when viewed from any angle (a circle).

6

ok but i mean

you use

Graphics.DrawMeshNow(censorMesh.Mesh, censorMesh.Transform.localToWorldMatrix);

when there is more then one correct ?

and

Graphics.Blit(source, destination); when its one ? and this works ?

maybe the censorMesh.Transform.localToWorldMatrix is flipped ?

7

More than one what? I’m a little confused what you are asking. I think you are getting caught up on my safety check in the code. The code always goes down the if branch, not the else branch since the script is disabled if there are no censor meshes. So the code might as well be:

private void OnRenderImage(RenderTexture source, RenderTexture destination)
{
    RenderTexture downSampledScreenCap = GetTempRenderTexture();  //scaled down by 8 times

    //down sample a smaller version to pass to shader
    //(If I were to save 'downSampledScreenCap' as a png now,
    //it would be right side up)
    Graphics.Blit(source, downSampledScreenCap);

    //pass the full image through the pipeline
    //(If I were to save 'destination' as a png now,
    //it would be right side up).
    Graphics.Blit(source, destination); 

    //pass the scaled down image to the shader, and render the
    //censor meshes with this material.
    _censorMaterial.SetTexture(RENDERERD_SCENE_PROPERTY_NAME, downSampledScreenCap);
    _censorMaterial.SetPass(0);
    foreach (CensorMesh censorMesh in _censorMeshesToDraw)
    {
        Graphics.DrawMeshNow(censorMesh.Mesh, censorMesh.Transform.localToWorldMatrix);
    }

    //If I were to save 'destination' as a png now the original part
    //of the image (the image before censor meshes were drawn) would
    //right side up, but the texture used for the censorship meshes
    //('downSampledScreenCap'), is upside down.

    RenderTexture.ReleaseTemporary(downSampledScreenCap);
}

This is basically the exact code that is executed on OpenGL and everything works fine there. The problem only happens on DirectX if there is another image effect script on the main camera that executes after this one. It has nothing to do with the number of meshes in my _censorMeshesToDraw list.

The localToWorldMatrix is correct because the mesh appears on the correct spot on screen. It is just the texture that it samples from is flipped.

8

The matrix contains scale info as well . maybe on directx the Y scale is flipped

IM don’t get what the censor mesh is … are u making this yourself ?

9

You are correct that the matrix contains scale information for the vertices on the mesh; however, my shader does not use UV’s on the vertices, so it wouldn’t matter if they were flipped since it calculates UV’s from the screen space position of pixels.

Though out of curiosity, I did a little test by using an asymmetrical mesh on the Y-axis (a mesh for a table) instead of a sphere. On DirectX, the table mesh was rightside up, just like in OpenGL when rendered with my effect, so this tells us that DirectX will not change the Y scale in this circumstance. (Of course, the texture on the table mesh was still upside down, just like with the sphere.)

CensorMesh is just a small class I made that contains a reference to the mesh that we will use for the censorship effect, as well as the transform to use to get the localToWorldMatrix. Basically, it holds all the data I need to manually draw the meshes without using Unity’s MeshRenderer. The reason why I can’t use MeshRenderers is because I want the whole scene to be drawn without censorship, then in OnRenderImage (which only happens once Unity has finished rendering from the given camera), I can manually pixelate certain parts of the image.

10

First, add “Cull Off” to your shader pass. If it is actually an image effect using blits quad.

Secondly; didnt examine your code throughly but you are using strange ways there, try to follow how unity does things.

Finally; here is a common vert function(certainly you want to calculate such stuff per vertex not per pixel) to handle halfpixel offset/d3d flipping (using unity provided structs):

            v2f_img vert(appdata_img v) {
                v2f_img o;
                o.pos = mul(UNITY_MATRIX_MVP, v.vertex);
                #ifdef UNITY_HALF_TEXEL_OFFSET
                    v.texcoord.y += _MainTex_TexelSize.y;
                #endif
                #if SHADER_API_D3D9
                    if (_MainTex_TexelSize.y < 0)
                        v.texcoord.y = 1.0 - v.texcoord.y;
                #endif
                o.uv = MultiplyUV(UNITY_MATRIX_TEXTURE0, v.texcoord);
                return o;
            }

hope it helps.

EDIT: For full screen pixelation effect, you can check my Aubergines Postprocess Shaders pack, for Per-Object pixelation effects, you can check my Pixelate Per-Object pack(which is what you are trying to do actually) in the assetstore.

11

So what you’re trying to do is pixelate certain areas of screen ?

12

That doesn’t fix the bug (I wouldn’t really expect it to), but I guess it makes sense to have that anyways though since there is no reason to do culling tests if we are drawing two triangles that always face the camera.

Are you talking about my shader code or the image effect code? Can you elaborate on what you think looks strange?

Actually, I do not want to calculate the UVs per vertex in this case. If I calculate the UVs from the screen space position in the vertex shader, then let the rasterizer interpolate the UVs, the sphere will appear to be a 3D object that is textured with a blurry portion of the image of the scene. If I calculate the screen space position per vertex, then let the rasterizer interpolate that, then calculate the UVs in the pixel shader, the sphere will appear as a completely flat blurry circle in the final image.

That aside, even if I do put the UV calculation in the vertex shader (with Unity’s suggested “fix”), the bug still persists.

Yes.

13

I personally had trouble implementing any Shader or UV solution to this, so I did the following (copy/pasted from a question of mine):

My fix for upside-down rendering when using DirectX: I was using OpenGL because one of my shaders was interacting with DirectX in such a way that it was rendered upside-down. I had trouble implementing the popular fix for this because of my unfamiliarity with Shader coding, so my way was to simply use OpenGL instead. This caused lighting problems, so the way I’ve since worked around it using DirectX is by using a RenderTexture on my main camera, and pointing an orthographic solid color Second Camera at a plane equipped with this RenderTexture (material w/ unlit shader) - that fixed my lighting issue and it also caused my image to no longer be flipped upside down.

14

From my experience this happens with render textures. So if you’re doing a post processing effect and mixing render textures with the source image for compositing like I was, then you only want to flip the y for the render texture coordinate.

v2f vert(appdata v)
    {
        v2f o;
               
        o.vertex = mul(UNITY_MATRIX_MVP, v.vertex);
       
        // store non-render texture uvs as is
        o.uv = TRANSFORM_TEX(v.uv, _MainTex);
        o.uvRT = o.uv;

        // flip the y if a render texture is being used
        if(_MainTex_TexelSize.y < 0)
        {
            o.uvRT.y = 1.0 - o.uv.y;
        }

        return o;
    }
15

Did you mean that, if a Graphics.Blit involves more than one textures as “source”, and if those additional textures are render textures binding to the pipeline by material.SetTextures, then their uv mappings should be manually flipped?

What if the source texture itself is actually of type RenderTexture, would it be affected then?

I’m having similar issue on Metal right now, even more strange is that my shaders works as expected on both OpenGL and Direct3D