Jaggy white pixel artifacts on edges with specular and antialiasing

Unity Engine
1

Here’s a problem.
You’ve got a mesh. You apply specular shader. You enable antialiasing in quality settings.
And suddenly a white pixel dots appear on some random edges when you zoom out from object.

2081830--136055--upload_2015-4-23_17-10-8.png
2081830--136055--upload_2015-4-23_17-10-8.png970×629 127 KB
2081830--136056--upload_2015-4-23_17-10-46.png
But this is not the problem itself. This is just a prehistory.

And now, here’s the actual problem: this is a known issue. And it’s known since Unity 3!

Many people confirmed it. And it happens regardless of using normal maps or even specular maps at all! You can see it in my screenshots above.

It. Just. Happens. With no relation to mipmaps, normal-mapping or filtering.

In previous Unity versions (3 and 4) there was a dirty hack which fixed it. This hack is more related to some voodoo magic rather than predictable behavior. It’s about adding this line in the end of your surface function:

o.Normal = fixed3(0,0,1);

i have no idea why it worked (and looks like nobody knows) but it worked.
Until now.

Since Unity 5 was released this fix stoped working.

So… I faced this issue. Again. With the only possible way to fix it: to find where it comes from.

You may think this is just another topic with some random guy raging about some bug, but it’s not the case.
I’m here to share the solution.

After several hours of trial and error, I finally found where it comes from. These white dots are caused by normal, which becomes non-normalized after transforming from tangential space (in which it’s described in surface function) to the space in which lighting function works (I don’t remember if it’s object space or world space).
So, in short, even if your normal is normalized in surface function, it may and will become non-normalized when it’s passed to lighting function.

So the only “nice and clean” solution is writing your own lighting function, in which you normalize incoming normal first.
For those of you who’s not so familiar with writing your own lighting functions… here it is:

#pragma surface surf NormalizedBlinnPhong halfasview
fixed4 LightingNormalizedBlinnPhong (SurfaceOutput s, fixed3 lightDir, fixed3 halfDir, fixed atten)
{
    // TODO: conditional normalization using ifdef
    fixed3 nN = normalize(s.Normal);
  
    fixed diff = max( 0, dot(nN, lightDir) );
    fixed nh = max( 0, dot(nN, halfDir) );
    fixed spec = pow(nh, s.Specular*128) * s.Gloss;
  
    fixed4 c;
    c.rgb = _LightColor0.rgb * (s.Albedo * diff + spec) * atten;
    UNITY_OPAQUE_ALPHA(c.a);
    return c;
}

Yes, you’re welcome. :wink:

It’s based on MobileBlinnPhong from “Mobile/Bumped Specular” and, obviously, this is old-style (pre-U5) surface function (with no realtime GI support).

Feel free to comment if this helped and if I found the right place where subj comes from.

The big question is still there, however. It’s:
Why do normals become un-normalized when AA is enabled in Quality Settings?

Hoping to get answer from UT in the comments below.

2 Likes
2

http://wiki.blender.org/index.php/Dev:Shading/Tangent_Space_Normal_Maps

It has the answers your looking for, but in short, transforming from tangent space will in general cause loss of normalization.
In most cases this is not noticeable…

3

I guessed that.
The only reason why loss of normalization can occur after any transformation is if transformation matrix is composed of non-normalized vectors. So in our case we may have all of them non-normalized (tangent, normal, binormal) due to interpolation from vertex program.

But it doesn’t explain why these white pixel artifacts appear only when AA is enabled.

When there’s no AA, the same vertex and fragment programs are used, with the same “loss of normalization due to interpolation” scenario. But they don’t cause these artifacts.

4

Perhaps they do cause these artifacts, but without the MSAA the nr of samples for those pixels is only 1, so chances of us getting a (even stronger) white pixel is a lot less.
It could also be that the parts getting so high pixelation would always be < 1 pixel otherwise, and thus blended away/discarded completly at those angles.
Oh and I have managed to get these pixely edges even without MSAA (deferred mode).

5

I’ve seen artifacts like this when building to mobile(iOS specifically) when I have split edges in my models. Interestingly not on UV seams, but on unwelded edges, which always struck me as odd since Unity will split your edges for you for smoothing group changes. I wonder if this has been the real cause.

6

Umm, generally due to mipmapping I thought it was best to HAVE a UV seam where you have a hard (unwelded?) edge. As it allows the painted area to be stretched out around it to prevent mip-mapping artifacts.

7

Yes, these artifacts could be caused by lack of UV padding.
But as you can see from the screenshots, they’re also present when no textures are used.

8

It’s not a hard edge thing, if the normals are still averaged(smoothed) the problem can still be seen. Even still, without a UV seam you should see no mipping artifacts, for the same reason why dilation helps mipping artifacts.

9

Hi, I’ve come across this same issue (checked and is not related to UVs in my case), and currently I’m using the standard shader, so I’ve spent some time trying to figure out where is the normal that I’ve to normalize.
I’ve checked the following files, and normalized every normal I could find (I don’t know if that’s the proper way to do it:P):
UnityGlobalIllumination.cginc
UnityPBSLighting.cginc
UnityStandardCore.cginc
UnityStandardCoreForward.cginc

But doesn’t seem to be working, so I wanted to ask if someone had to solve this same problem that I’m having with the standard shader.
Cheers.

(also here is a screenshot of my problem)
2912097--214731--Screenshot_3.png

10

@geroppo Are you sure it’s about the same issue discussed here?
From your screenshots I see the issue appearing only in the shadow area. And seems like it’s not getting white, but is getting the main brownish color instead. As if in those artifact areas there were no shadow for some reson.

Try to disable shadows at all. Do you still see the problem?

11

you were right, seems to be another issue. Disabling shadows (turning of the “receive shadow” of each mesh) makes the artifacts dissapear. Any guess what could it be then ? because it seems to be related to anti aliasing, if I turn it off then it gets fixed. I’ve tried using the post processing effect anti aliasing that unity provides, but doesn’t solve the original issue (attached below)
2912122--214735--Screenshot_2.png .
THAT^ get’s fixed only by using the um…“built in” anti aliasing (instead of the post processing one). But after turning on that anti aliasing, the white/shadow artifacts comes up.

EDIT: Also, in the UnityGlobalIllumination.cginc at the end of the function DecodeDirectionalSpecularLightmap I found a comment saying something (hopefully) related to these jagged lines. Code below

inline half3 DecodeDirectionalSpecularLightmap (half3 color, fixed4 dirTex, half3 normalWorld, bool isRealtimeLightmap, fixed4 realtimeNormalTex, out UnityLight o_light)
{
    o_light.color = color;
    o_light.dir = dirTex.xyz * 2 - 1;

    // The length of the direction vector is the light's "directionality", i.e. 1 for all light coming from this direction,
    // lower values for more spread out, ambient light.
    half directionality = max(0.001, length(o_light.dir));
    o_light.dir /= directionality;

    #ifdef DYNAMICLIGHTMAP_ON
    if (isRealtimeLightmap)
    {
        // Realtime directional lightmaps' intensity needs to be divided by N.L
        // to get the incoming light intensity. Baked directional lightmaps are already
        // output like that (including the max() to prevent div by zero).
        half3 realtimeNormal = realtimeNormalTex.zyx * 2 - 1;
        o_light.color /= max(0.125, dot(normalize(realtimeNormal), normalize(o_light.dir)));
    }
    #endif

    o_light.ndotl = LambertTerm(normalize(normalWorld), normalize(o_light.dir));

    // Split light into the directional and ambient parts, according to the directionality factor.
    half3 ambient = o_light.color * (1 - directionality);
    o_light.color = o_light.color * directionality;

    // Technically this is incorrect, but helps hide jagged light edge at the object silhouettes and
    // makes normalmaps show up.
    ambient *= o_light.ndotl;
    return ambient;
}
12

Sorry man. I don’t know where it comes from. But, obviously, it’s another issue. Which could be (and probably is) completely unrelated to the one discussed here.

So the best thing I can tell is to try to find the answer yourself. It’s Unity, get used to it! :wink: It’s all about constantly fixing some bugs after UT that appear in unexpected places and got fixed even more unexpected way. Especially when it comes to graphics/shaders.

You could just aplly trial and error aproach here, top-to bottom.
Modify 31st line replacing o_light.ndotl with some non-critical-point constant (i.e., not perfect 0 or 1 or 0.5 but something in-between, like 0.738).
And then, if the problem is in ndotl value, check what you’ve got there and dig deeper to find where it comes from, step-by-step

13

Alright, thanks!

14

And send a bugreport with a simplified-isolated-case scene, of course. Don’t expect it to be fixed soon, though.
But there’s a really high chance that your bugreport today will become the actual reason why this specific bug will be fixed two years later.

15

I leave this here in case someone else comes here having my very same problem, it seems to be an issue that has already been reported (3 years ago…):

16

This will be “fixed” likely in either 5.6 or 5.7 when they add a new Forward+ render path that doesn’t use the screen space shadows.

Until then I use this:
https://forum.unity3d.com/threads/fixing-screen-space-directional-shadows-and-anti-aliasing.379902/

2 Likes
17

Oh wow thank you very much!

18

Is there any fix for this problem? I encountered on 2018.11.f1.
I’m using standard shader with no textures in the example. I’m not using shadows in the scene.

1727×505 179 KB

19

Looks like what happens with either bad geometry or otherwise extremely thin bits of geometry with vertex normals pointing in significantly different directions.

It’s a little complicated to explain, but the way MSAA works it’s possible to end up drawing a triangle using values that aren’t on the triangle.

To explain that, you need to understand rasterization and barycentric interpolation.

A triangle is made up of three vertices. When drawing a triangle on screen, the values used are an interpolation of the three vertex values based on their distance from each traingle within the triangle.
https://www.scratchapixel.com/lessons/3d-basic-rendering/ray-tracing-rendering-a-triangle/barycentric-coordinates

The GPU draws triangles using rasterization. The short version is the GPU calculates if a triangle covers the center of each pixel or not, aka a coverage sample, and if it does, it draws the triangle using the interpolated values for that position.

Where MSAA causes problems is it does multiple coverage samples within a pixel to see if it hits a triangle (none at the center of the pixel), but it still uses the interpolated values from the center of the pixel. This means the MSAA coverage samples may determine a triangle is visible to a pixel, but the triangle may not actually cover the center of the pixel. This means it’ll interpolated values outside of the triangle! This is a problem for triangles with a decent amount of normal variation as that over interpolated value may be pointing in a direction that doesn’t exist in the vertex data.

One solution is to not have super thin bits of geometry, likely using mesh LODs to remove as many of these as possible. This isn’t perfect as you can get thin geometry just from rotating a mesh so that some faces are being viewed edge on from the camera view.

The other solution is to use centroid sampling on the vertex to fragment interpolators. This requires a custom shader, and can make visually smooth continuous surfaces have seams. It’s also been broken in Unity for the last 5 years such that it flickers on and off for reasons unknown.

1 Like
20

Curious what you mean here by centroid sampling potentially causing seams, could you explain/elaborate?