Highest quality AA for raytraced shots


Sorry if this has been asked before, but I couldn't seem to find a thread on it in the VP section at least:

is there a current consensus on what antialiasing solution yields the "best" results (as in highest visual quality with the least amount of artifacting) when rendering out frames using raytracing in HDRP? As these sequences are being rendered out as exr's using Recorder, performance is not really that much of a concern.

TAA seems to lead to significant ghosting, especially visible in situations where we don't include motionblur (e.g. for lookdev turntables). So far testing the other two, while they don't produce the same ghosting issue, they seem to have their own set of drawbacks, such as an overall softening of the image. Definitely open to the possibility that my settings are partly/fully to blame.

Is CTAA seen as a viable solution? Seems to be brought up a lot when discussing AA in Unity, and the examples on the store page do look pretty decent for sure.

Totally get that every solution probably has its own set of pros and cons. Dusty old vfx artist here trying to wrap my head around how everything works over here in unity realtime world, so my apologies if you end up having to explain it to me like I'm five : )



In a new Recorder version (4.0.0-pre.3-4), we introduced a concept of sub pixel jittering when you enable accumulation as well as shadow map filtering for Spot lights if I remember right. The shadow map filtering will not add artefacts, but the jittering might (might make the image slightly softer).
This fixes certain aliasing problems but not all.

If this is still not enough, you could render at a higher resolution and downscale outside of unity with an appropriate filter (Lanczos or bicubic?).

Hope this helps...

I find none of the cam's AA gives me good results.
Instead, I use DLSS, purely for AA. So its enough to simply turn on DLSS and use high quality preset.

You mentioned VFX, so then it's likely you'll be rendering non fullframe CGI, ie you'll be comping over some sort of backplate. Unfortunately in that case, there is no solution as Unity has a bug whereby no AA happens at all against the background, when you need to save the frame with the background as an ALPHA.

In other words, the AA is fine everywhere, except on the outer edges of geometry where it touches the background. You can't tell by simply looking at the frame in Unity. However when you open the resulting EXR and inspect the alpha, it becomes very clear what the issue is.
You can somewhat do a workaround of the issue, by doing a 2nd pass with a custom override, and apply an unlit white shader to everything, thereby giving you a black and white image. This custom pass seems to work somewhat better, however the issue is the need to do it in the 1st place.

For more info on the issue, see here:

@vladala thank you for your reply! I was not aware that accumulation was used for anything other than motion blur and path tracer sampling convergence, so that is great to know. And as I've come to understand, the ghosting issues I'm having with TAA is inherent to how it works, especially when objects move and reveal either new parts of their own geometry or whatever was occluded by the object in the previous frame. I'll give the accumulation settings a try and see if I can get results that are acceptably comparable to upres render/downscale in comp.

@newguy123 thank you so much for this, I would definitely have ended up down the same rabbit hole as you trying to figure out what was going on with those alpha issues, pulling my hair out in the process. Your thread and all the testing you did probably saved me a ton of work (and increased blood pressure), so thank you! : )

Sounds like that for now, especially on non full frame cg shots that need to integrate with a plate, I might just be better off running double res frames with no AA from Unity, doing the antialiasing and downscaling in Nuke and waiting for an improved AA solution in Unity (guessing addressing the alpha issue in DLSS is not something nvidia would put especially high on their prio list). Your second pass DIY alpha pass also sounds like a decent workaround @newguy123 , will definitely look into that as well!

Might also check out CTAA while its on discount, but no idea if that also introduces issues with alpha if it is doing some type of fancy postprocess sharpening to the image to get those results.

Thanks again to you both for the insightful replies, definitely very enlightening!

Ah @larsStranden you misunderstood me I think. The lack of AA against background is a general Unity bug and present no matter what you use for AA

That being said, they made some changes recently to TAA in the latest 2023 alpha or 2022.2 beta, not sure if it will be backported to the current LTS or not...

Oh ok, I see, thanks for clarifying! I think I picked that up from one of the dev replies to your posts in the thread you linked, where it sounded like DLSS alpha support was the issue and not something they could fix internally (due to it being an nvidia thing)

Here's some info on the coming changes to TAA that I mentioned earlier:


fantastic, thank you again. I'll keep an eye on that thread, the examples they post using the updated TAA implementation at least seem pretty promising as far as the ghosting issue goes

1 Like