RenderTexture with clear flags set to depth has weird outline (Anti-Aliasing issue)

Unity Engine
1

Hi, so I’m rendering out a 3D mesh to a Texture2D and drawing that texture in an editor window but when I turn on depth clear flags so that I can have a transparent background, there is a weird outline around the mesh - is there anyway to fix this?

Below are two screenshots - left is with clearflags set to solidcolor (no weird outline) and right is set to depth (has weird outline)

Also below that is the relevant section of code

Thanks :slight_smile:

RenderTexture rt = new RenderTexture(renderSize.x, renderSize.y, 24);
rt.Create();

previewRenderer.camera.targetTexture = rt;
previewRenderer.camera.Render();
       
var currentRT = RenderTexture.active;       
RenderTexture.active = rt;
Texture2D render = new Texture2D(rt.width, rt.height);
render.ReadPixels(new Rect(0, 0, rt.width, rt.height), 0, 0);
render.Apply();
RenderTexture.active = currentRT;

//render is then drawn in the UI
2

They both have outlines really. When you set the clear to depth only, really you’re saying “use the default clear color of (0,0,0,0)”. This means the background is “clear” because the alpha defaults to fully transparent, but you’re rendering using MSAA, so the anti-aliased edges are a blend of the black default clear color and your rendered object. The short version is the colors in the final image have been premultiplied by the alpha.

The easiest solution is to render the image using a premultiplied alpha shader… of which Unity has none built in that actually work properly for the simple case of using an input texture that has premultiplied alpha. Like this:

Shader "Custom/Premultiplied Alpha Blend"
{
    Properties
    {
        _MainTex("Tex", 2D) = "white" {}
    }
    SubShader
    {
        Tags { "Queue"="Transparent" "PreviewType"="Plane" }

        LOD 100

        ZWrite Off
        Blend One OneMinusSrcAlpha

        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
           
            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f
            {
                float4 pos : SV_POSITION;
                float2 uv : TEXCOORD0;
            };

            sampler2D _MainTex;
           
            v2f vert (appdata v)
            {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                o.uv = v.uv;
                return o;
            }
           
            fixed4 frag (v2f i) : SV_Target
            {
                return tex2D(_MainTex, i.uv);
            }
            ENDCG
        }
    }
}

You can use that along with something like EditorGUI.DrawPreviewTexture to draw a texture with a custom material rather than just the default alpha blending.

The other options would be to disable MSAA and filtering on your texture, or to use a post process on your images to divide the color values by the alpha to undo the premultiplication, and then dilate the color into the fully transparent areas to ensure filtering doesn’t still produce a dark edge.

3

Thanks for that, I’ll give it a go - hopefully should do the trick :smile:

4

Ok so I tried using the custom shader but didn’t seem to change anything so maybe I did it wrong?
Here’s a screenshot (above is new method, below is old)

And here’s the code for drawing it (rendersList is a list of Texture2Ds - converted from RenderTextures as in my original post)

if (rendersList.Count > 0)
{
     Rect r = GUILayoutUtility.GetRect(128, 128);
     r.width = 128;
     EditorGUI.DrawPreviewTexture(r, rendersList[0], new Material(Shader.Find("IconGenerator/ImgShader")));
}
5

bleh … obviously something isn’t working as there’s almost no difference between those two images.

One random option. Try setting the camera to clear to a solid color, and set that solid color to 0,0,0,0. Otherwise I wonder if custom blend modes are broken for DrawPreviewTexture. :confused:

6

Still no luck :frowning:

I also tried setting it to 1,0,0,0 and this is what I got so there is some difference?

also, if I turn off AA for the camera it gets rid of the outline as expected, from here is there then a way to do some form of AA/smoothing that wouldn’t make the outline?

7

So, Unity has a long running bug that it displays textures in the UI using the wrong gamma. A small change to the shader should make the original setup work.

Shader "Editor/Premultiplied Alpha Texture"
{
    Properties
    {
        _MainTex("Tex", 2D) = "white" {}
    }
    SubShader
    {
        Tags { "Queue"="Transparent" "PreviewType"="Plane" }

        LOD 100

        ZWrite Off
        Blend One OneMinusSrcAlpha

        Pass
        {
            CGPROGRAM
            #pragma vertex vert
            #pragma fragment frag
         
            #include "UnityCG.cginc"

            struct appdata
            {
                float4 vertex : POSITION;
                float2 uv : TEXCOORD0;
            };

            struct v2f
            {
                float4 pos : SV_POSITION;
                float2 uv : TEXCOORD0;
            };

            sampler2D _MainTex;
         
            v2f vert (appdata v)
            {
                v2f o;
                o.pos = UnityObjectToClipPos(v.vertex);
                o.uv = v.uv;
                return o;
            }
         
            half4 frag (v2f i) : SV_Target
            {
                half4 col = tex2D(_MainTex, i.uv);
                // gamma correction for use in the editor UI
                col.rgb = LinearToGammaSpace(col.rgb);
                return col;
            }
            ENDCG
        }
    }
}

edit: nope, it’s still wrong. Closer, but now it has a bright edge. :confused:
Drawing textures in the editor properly is a huge pain.

8

Left: Before, Right: with correction
4462186--409570--upload_2019-4-23_12-59-14.png

9

Could I see your code for the editor script because mines still not quite right :confused:

thanks for all the help btw

10

The one thing that did kinda work was rendering the texture at twice the resolution with MXAA turned off, then using EditorGUI.DrawPreviewTexture and drawing at the original size (essentially downscaling it and doing the AA for me)

only issue is I need to be able to save as png but obviously all I have available is the double scale texture2d - not the nicely scaled down one that shows in the UI

11

Okay, solved it a different way. Go back to the previous shader without correction. The trick is to set the proper sRGB settings for the render texture (sRGB) and the Texture2D (linear).

using UnityEngine;
using UnityEditor;

[RequireComponent(typeof(Camera))]
public class CameraIcon : MonoBehaviour
{
    public enum IconResolutions
    {
        x64 = 64,
        x128 = 128,
        x256 = 256,
        x512 = 512
    }

    public IconResolutions iconResolution;
    public Texture2D icon;

#if UNITY_EDITOR
    public void RenderIcon()
    {
        var cam = GetComponent<Camera>();
        int res = (int)iconResolution;

        var rtd = new RenderTextureDescriptor(res, res) { depthBufferBits = 24, msaaSamples = 8, useMipMap = false, sRGB = true };
        var rt = new RenderTexture(rtd);
        rt.Create();

        cam.targetTexture = rt;
        cam.Render();
        cam.targetTexture = null;

        if (icon == null)
        {
            icon = new Texture2D(res, res, TextureFormat.RGBA32, false, true);
        }
        else if (icon.width != res)
        {
            icon.Resize(res, res);
        }

        var oldActive = RenderTexture.active;
        RenderTexture.active = rt;
        icon.ReadPixels(new Rect(0, 0, res, res), 0, 0);
        icon.Apply();
        RenderTexture.active = oldActive;

        DestroyImmediate(rt);
    }
#endif
}

#if UNITY_EDITOR
[CustomEditor(typeof(CameraIcon))]
public class CameraIconEditor : Editor
{
    private Material iconMat;

    public override void OnInspectorGUI()
    {
        base.OnInspectorGUI();

        var ci = (target as CameraIcon);

        if (GUILayout.Button("UpdateIcon"))
            ci.RenderIcon();
        if (ci.icon != null)
        {
            if (iconMat == null)
                iconMat = new Material(Shader.Find("Editor/Premultiplied Alpha Blend"));
            Rect rect = EditorGUILayout.GetControlRect(false, (int)ci.iconResolution);
            rect.width = rect.height;
            EditorGUI.DrawPreviewTexture(rect, ci.icon, iconMat);
        }
    }
}
#endif
12

Yep that seems to work, now I just need to get it to work with the PreviewRenderUtility - hopefully shouldn’t need to change much :stuck_out_tongue:

13

Ehh, still has the border for mine though :confused:

It’s not a scene camera, its PreviewRenderUtility but I thought this would work

Test t = previewRenderer.camera.gameObject.AddComponent<Test>();
t.iconResolution = Test.IconResolutions.x512;
Texture2D render = t.RenderIcon(); //I made the function return the texture

Then

Material iconMat = new Material(Shader.Find("IconGenerator/ImgShader"));
Rect rect = EditorGUILayout.GetControlRect(false, (int)previewSize);
rect.width = rect.height;
EditorGUI.DrawPreviewTexture(rect, rendersList[0], iconMat); //rendersList[0] is the same as the render Texture2D above

1461×513 84.9 KB

Any thoughts?
Might just rewrite my thing to avoid using the PreviewRenderUtility altogether - could just instantiate a camera in scene view, render the image, then delete it straight away

14

Ok so I did what I said at the bottom of my last comment and ditched PreviewRenderUtility and it works if I use EditorGUI.DrawPreviewTexture with the custom shader however I need to be able to save the render as a png (and really just need it as a Texture2D as I am drawing a SelectionGrid which takes Texture2Ds) - is there anyway to save the render with the shader applied because otherwise the saved image just looks like the bottom pic rather than the top

15

The shader isn’t modifying the data, it’s modifying how the data is displayed. If you save the texture2D to a png, then load that png and display it with the same shader it should work exactly the same. Unity’s PNG save and load shouldn’t be modifying the data at all.

The next “step” I would take if you want to try to avoid some of this pain is the “undo the premultiplied alpha” step I mentioned above. This could be done with a blit, or by modifying the data on the CPU before saving it to disk.

Color[] pixels = icon.GetPixels();
for (int i=0; i < pixels.Length; i++)
{
    if (pixels[i].a < 1f && pixels[i].a > 0f)
    {
        pixels[i].r /= pixels[i].a;
        pixels[i].g /= pixels[i].a;
        pixels[i].b /= pixels[i].a;
    }
}
icon.SetPixels(pixels);
var bytes = icon.EncodeToPNG();
// etc
16

Issue is that I need to be able to view the image without using any special shaders, like in other programs or windows photo viewer for example. Is there anyway to sort of bake the shader into the data if that makes sense?

17

… :eyes:

See my previous post which you quoted?

18

Sorry I somehow missed that whole bottom part, thanks I’ll take a look at that when I’m home tomorrow :slight_smile:

19

Okay I got a chance to try it out and IT WORKED!
Seriously dude, thanks for all your help, I’m really grateful for it :smile:

20

Sorry for necro posting a bit, but I was wondering if you could elaborate on this?
Using this code it improves the result for sure, but doesn’t quite remove it. It goes from this:

To this

This is using GUI.DrawTexture, and white using EditorGUI.DrawPreviewTexture with the material works, it also will no longer respect GUI masking (like in scroll views).
Of course using point filtering will solve this, but that makes it quite pixel (duh).
Any further recommendations would be appreciated. :slight_smile: