MSAA has two achilles’ heels. HDR and over interpolation, and this use case is hitting both.
The HDR issue is fairly well documented elsewhere by other people and has been known about for a very long time. Unfortunately it’s also an issue that many choose to ignore, in part because it can be relatively costly to solve. This long overview of how MSAA works has an example of the issue in the section on MSAA Resolve:
Specifically these two images:
Both of these images are using MSAA, but the first one looks aliased still. The problem is in how MSAA stores “multiple samples”, and how it resolves the image from an multisample render target to an non multisample texture for post processing or display. The article above goes into great detail on both of these, but the short version is the multisample render target has N color values per pixel where N is the number in the “Nx Multi Sampling” option. We’ll get a bit into how each color value is determined in a moment. But for simplicity lets imagine a 2x MSAA rendering of the edge of a wall with a bright HDR sky behind it like in the above examples, those color values are the linear color values. The wall might be a “0.2” and the sky might be “4.0”. To resolve the 2x MS render target to a texture that can be displayed, the GPU doesn’t do anything particularly fancy. It takes the average of those two values and displays that. For values that aren’t HDR, this results in a nice useful value to display that results in useful anti-aliasing. For the example described above you get a value of “2.1” … way over 1.0, so you get what appears to be no anti-aliasing because the value is already being clipped.
The second image gets around this by determining the LDR (low dynamic range) value of each individual color value before doing the resolve. In the areas that are super bright, it clamps those to 1.0, and then the resolved value isn’t blown out. So you get a visible transition from the “0.2” wall to the “4.0” sky with a displayed “0.6” in between.
The downside to this approach is you lose the HDR values after resolving. More advanced methods do what’s known as inverse tone mapping, which lets you retain the HDR values after the resolve.
This one is hard to fix in Unity because Unity didn’t support directly sampling an MSAA target until quite recently, and AFAIK still don’t themselves do any attempt to do custom resolves. They use the basic hardware resolve on the render target and then rely on some kind of post process anti-aliasing (like FXAA or TAA) to clean up whatever is left.
The second issue I mentioned, over interpolation, comes down to the difference between MSAA and SSAA (super sample anti-aliasing). Both MSAA and SSAA work by effectively rendering multiple positions within each pixel. Super sample anti-aliasing is roughly equivalent to rendering at a much higher resolution and downscaling the image to the display resolution. This is expensive, so MSAA was created to offer a cheaper alternative. What MSAA does is it renders only the depth values for multiple positions within each pixel, known as a coverage sample. The color values are still only rendered at the center of the pixel. How this works out is if a triangle is visible in one or more of the coverage samples, the triangle is rendered once at the center of the pixel and that one color stored in the correlating color samples for those coverage samples. The problem comes when a triangle covers only one coverage sample, but isn’t visible at the center of the screen, the GPU interpolates the vertex values all the way out to the center of the pixel to render. For UVs this works out pretty well. For vertex normals this can cause the normal value to flip, meaning your vertex normal is suddenly pointing away from the camera. In the case of something like a rim light that can mean the color value gets much brighter than it would be normally. And as described above, bright values and MSAA don’t get along, so you suddenly get aliasing where you didn’t before!
For the second issue, you can do a few things to work around it. One is to clamp the normal (or potentially other interpolated values if need be) so they don’t go beyond useful values. There’s also a thing called centroid sampling where the position used for rendering the color value isn’t at the center of the screen, but instead at the center of the coverage samples. This issue in particular got a lot of attention from people working on VR, and Valve did a talk on doing just this. I implemented this and another technique for minimizing edge aliasing for the Standard shader here:
For the HDR aliasing issue, the above shader may solve it enough to not be as noticeable for your use case. But others have written shaders that apply an inverse tone mapping when rendering the objects in the scene . This can cause some issues for alpha blending, but otherwise is very effective. (Note, the thread above dates back to a time when Unity didn’t support MSAA at all if HDR was enabled, which is no longer the case.) Also as mentioned above, FXAA or better yet TAA can help clean up a lot of the issues, or be used in total absence of MSAA. Modern AAA games almost all use TAA exclusively at this point unless they’re for VR, and even then many VR only titles only use some form of TAA. TAA doesn’t have the same issue as MSAA as it is inherently doing a “custom resolve” of multiple rendered frames and can do something akin to inverse tone mapping.