[Open Source] FidelityFX Super Resolution 3.1 (FSR3) Upscaler for Unity (BiRP, DX11, PPV2, HDRP17, multi-platform)

[Astfgl]

[GitHub]

Hey all,

These past few months I have been working independently on researching FidelityFX Super Resolution 2 (or FSR2 for short) and seeing if I could make it work within Unity. My goals were rather specific: I wanted to make it work with Unity’s built-in render pipeline, Post-Processing Stack V2 and the older DX11 graphics API, as well as on Mac, Linux and consoles.

When starting out I had no idea if this was possible at all. After all, FSR2 is generally associated with more modern PC graphics APIs (DX12 and Vulkan) and more advanced rendering pipelines. My particular needs were the driving force for how I approached this project and they called for some unorthodox solutions. I was half-expecting to run into some insurmountable roadblocks and having to give up halfway through, but to my pleasant surprise I managed to make FSR 2.2 work, in Unity, on the built-in render pipeline, with DX11 and even on Linux, MacOS, and both 8th gen and 9th gen consoles. And the results are rather spectacular:

Compared to the standard post-process anti-aliasing methods offered by Unity (FXAA, SMAA and TAA), FSR 2.2 produces a much sharper and cleaner picture, with better sub-pixel detail resolution, far less temporal shimmering and no unsightly ghosting artifacts in fast motion. In addition to that, because FSR2 upscales from lower internal rendering resolutions, you will see improved framerates even when compared to having no AA. This is a major boon for games that are GPU-bottlenecked, as it will improve both image quality and performance.

While I did work on FSR2 for Unity with a specific game in mind (more on that later), I developed this plugin as a standalone project with the intention of open sourcing it and allowing anyone to play around with it. And so that is what I’m doing now: the source code of FSR2 for Unity is available on GitHub right now, licensed under MIT meaning you are free to do whatever you like with it. More details on how this implementation works can be found on the GitHub page.

Right now this project only offers an integration with the built-in render pipeline. The core FSR2 implementation however is render pipeline-agnostic, so there’s nothing stopping anyone from building a URP or HDRP integration based on the same core classes.

I have also been testing this with Unity’s Post-Processing Stack V2 (PPV2) to make sure the two play well together. While this does work, the setup is not ideal yet: PPV2 currently has to run entirely before FSR2 which works fine for most effects, but ideally effects like bloom, depth of field and motion blur should be applied after FSR2’s upscaling. It still produces rather nice results though:

I have spent some time trying to adapt FSR2 into a PPV2 post effect, but it seems PPV2’s framework is a bit too rigid to make this work well. The way I’m looking at it now, I think the best way forward would be to modify PPV2 itself and customize it to make it upscaling-aware, to add FSR2 as an alternative anti-aliasing method, and to split the existing built-in effects into before- and after-upscaling steps. This is something of a project in itself, but I hope to look into this in the near future.

Update: Full integration with PPV2 is now available too. This is the recommended way of combining FSR2 with PPV2. More details in the post below.

As I mentioned, I worked on this project with a specific game in mind and I have been collaborating with its creators over the past couple of months to integrate FSR2 into that game, after demonstrating my initial results. I cannot divulge right now which game that is, but rest assured that this implementation is already being battle tested. Hopefully within the next few weeks I can tell you which game it is and I’ll also be able to show you better comparison pictures.

Update: The game in question is Isonzo, more details and screenshots in the post below.

Before anyone asks, I am aware of what The Naked Dev is working on and it is not my intention to ‘compete’ with or in any way undermine what they’re doing. I felt justified in working on my own project because my priorities (focus on built-in render pipeline, DX11, Mac, Linux and console support) did not appear to align entirely with what they’re committing to, at least not initially. I also couldn’t be sure if their approach to integrating FSR2 into Unity would work for the game I had in mind. You should see my project more as a tinkerer’s version of FSR2 for Unity; if you’re not afraid to get your hands dirty and adapt it to work for your particular application, then you might find a good starting point here. If you’re looking for something more plug-and-play, more polished and with a better guarantee of long-term support, then you will want to keep an eye on TND’s project. The way I see it, having more options available is never a bad thing.

Update: The FSR Upscaling asset from The Naked Dev migrated from a native plugin solution to make use of my open source tech shortly after its release, so their asset now combines the best of both worlds!

Have fun picking this apart! I’m looking forward to seeing all of your reactions.

[Dark-1-Games]

Major props for the amazing work!

[Astfgl]

Thanks!

As you could read in the other thread, Alterego and I have joined forces, which means that their asset will gain the multi-platform advantages from my implementation, while my work will benefit from their URP and HDRP integrations. This should lead to the best of both worlds: FSR 2.2 available for everyone, everywhere.

In the meantime I have also been testing on mobile platforms and, well:

It’s looking very promising. I’ll have to do a bit more research and testing before I can confidently say Android and iOS are fully supported, but it’s definitely working already.

[TheDigitalzombie]

Absolutely awesome! Thank you!

You already stole 10 hours from me and it’s still going. But thats because of my strange setup and not related to the implementation. Ah, I’ll get there …

[pwka]

Amazing job! Awesome that you made this available to everyone!

@Astfgl

my work will benefit from their URP and HDRP integrations

Does this mean that your open source solution will also get options for urp and hdrp?
Cheers!

[Astfgl]

@pwka

Amazing job! Awesome that you made this available to everyone!

Thanks, and you’re welcome!

Does this mean that your open source solution will also get options for urp and hdrp?
Cheers!

That is unlikely to happen anytime soon. Of course I don’t want to cut into Alterego’s sales too much, which would happen if I made open source URP and HDRP integrations. Besides that, my personal experience with URP and HDRP is very limited so making integrations for those would cost me a lot of time for research, time that I think is better spent elsewhere right now.

The way I’m looking at it now is: the open source project contains the backend of the FSR2 implementation, together with a single reference integration. Alterego’s asset will be the ‘product’ that has all the integrations, is more polished and user-friendly, and that will receive more professional long-term support.

[pwka]

Yeah, I still want to buy Alterego’s solution. Antialiasing in Unity is getting worse, and FSR is gamechanger in that regard. Thanks for the clarification!

[Astfgl]

While Alterego Studios have been finishing up their asset (congrats to them on the release!), I have been working on steadily improving the core FSR2 backend code, as well as creating a new integration for the built-in render pipeline. I already alluded to this in my original post:

I think the best way forward would be to modify PPV2 itself and customize it to make it upscaling-aware, to add FSR2 as an alternative anti-aliasing method, and to split the existing built-in effects into before- and after-upscaling steps.

I have now fully integrated FSR2 into Unity’s Post-Processing Stack V2 codebase and released it as a drop-in replacement package on the Github page. Source code is also available. This gets FSR2’s placement in the render pipeline much closer to the ideal picture suggested by AMD:

With this, post-processing effects that should get rendered at full resolution instead of being upscaled, such as bloom, motion blur, depth of field and film grain, are now applied after FSR2 upscaling. This means that previously problematic effects that caused noticeable artifacts are now looking correct. The integration itself also feels much more natural and easier to implement into a project.

I should note that while this PPV2 package is perfectly usable in its current state, it’s not trying to be everything for everybody. It still only supports the built-in render pipeline (URP support for PPV2 was dropped a long time ago), it doesn’t support VR (yet) and I’m sure there will be more features currently missing. If you’re using BiRP with an unmodified PPV2 package, then this should be easy to drop in. Otherwise, you should see this package as a better example of how to fully integrate the core FSR2 classes into a render pipeline.

Aside from that, the following work has been done on the core FSR2 implementation since my last post here in this thread:

• The FSR2 shaders have been updated to FSR 2.2.1

• The experimental auto-transparency & composition feature that was added in FSR 2.2 has been implemented.

• The issue where enabling auto-exposure caused a black screen on some platforms has been fixed.

• The ESRAM-related issue on Xbox One when auto-reactive mask is in use has been fixed.

• Various optimizations to reduce VRAM usage.

• Some refactoring to improve clarity and readability.

I’m also currently working on dynamic resolution support using ScalableBufferManager, and I hope to look into VR soon as well. At the very least, to get an idea of what is required to make VR work with FSR2.

[Astfgl]

Finally I am able to reveal which title I have been working on these past few months to develop and test this implementation of FSR2. The game in question is Isonzo, part of the WW1 Game Series. Yesterday the update with FSR2 integration was released on Steam. Consoles will follow soon.

Here is a comparison between No AA, FXAA, TAA and FSR2 Quality & Performance on the Cengio map:

I have been working as a programmer on the WW1 Game Series team for several years, where I was primarily responsible for producing the console versions. Upon release of Isonzo late last year, I was rather unhappy with the overall image quality and performance that we were able to achieve, particularly on consoles. Unity’s anti-aliasing solutions all made the game look worse and all other alternatives we looked into did not work for us for one reason or another. Meanwhile FSR2 had been open sourced shortly before and I knew that would be a great fit for our game, but with all my other responsibilities I simply did not have the time or energy to properly investigate it.

At the end of last year I ended my tenure on the WW1 team, with the intention of taking some time for myself to work on several research projects. Isonzo’s image quality remained a pet peeve of mine however, so properly researching FSR2 for Unity was one of the projects that I wanted to work on. When FSR 2.2 was open sourced in mid-February, that was the trigger for me to genuinely pick up this project and see if I could make it work. And the rest, well, is history.

Working together with the WW1 team to integrate FSR2 into Isonzo has really helped a lot to quickly get this FSR2 implementation up to speed. It gave me a concrete use case to test my work against, it confronted me with many real-world problems that my prototype projects didn’t show, and it gave me a lot of feedback to help fix bugs and to push me to keep improving the quality. It has made the core FSR2 implementation very solid and a good foundation to base other game integrations on.

The integration in Isonzo is based on the Post-Processing Stack V2 package that I released last week, or rather is backported from it to the older customized version of PPV2 that Isonzo uses. Initially I used the image effect-based integration but that did cause some graphical issues in places. Migrating to the PPV2-based integration cleared up the problems we had with effects like motion blur, and certain full screen overlays like screen flashing, blinking and blindness effects. That finally made the FSR2 feature feel production-ready.

[Astfgl]

Here is another comparison, this time on the Gorizia map:

FXAA simply blurs the image, while TAA smooths over a lot of the fine details. FSR2 by comparison keeps things crisp and sharp and brings out a lot more texture detail, even when compared to No AA. Also take note of thin geometry like grass and trees; FSR2 manages to reconstruct those very accurately, leading to much fuller-looking foliage than on any of the alternatives.

What’s not visible in these still shots is just how bad the ghosting artifacts are with TAA in motion. When the first-person weapon model moves across the screen, TAA produces an awful looking ghostly trail behind it that is very noticeable and distracting. FSR2 is not without its motion artifacts either, but instead of a ghost trail it will merely show some pixellation in places that are under-sampled, which is far less distracting. FSR2 also suffers a lot less from shimmering artifacts in motion compared to the other options.

(The forum only allows a maximum of 5 uploads per post, hence the need for this separate post.)

[baggyg]

Sounds very progressive on all fronts. Between yourself and Alterego, do you think we could see VR support any time soon?

[Astfgl]

I looked into VR a while back and quickly came to the conclusion that VR support in Unity is a complete mess that’s shoehorned into the different rendering and post-processing pipelines in a multitude of awkward ways. That makes it hard to know where to even begin with integrating FSR2.

I think the best way forward would be to generalize the texture array input/output support that I already added for HDRP some time ago, and then leaving it to each render pipeline integration to manage which buffers and array indices get passed to FSR2 and when. That means the core FSR2 dispatching code won’t really have to know about VR, it just gets some buffers that it needs to upscale. Any other approach would massively over-complicate the core FSR2 classes and make it that much harder to maintain.

Anyway, I guess what I’m trying to say is that VR support is not a trivial thing; it will require a bunch more research and each render pipeline and stereoscopic rendering variant will have to be considered separately.

[blaher]

This is awesome… though motion is rough on moving plants. I read the section about motion vectors… any links or info on how to go about “rendering motion vectors” or what you mean by this? is this something in hdrp only? any way to fix blurry motion/ghosted mess of plants w built in render pipe?

[blaher]

@Astfgl

I looked into VR a while back and quickly came to the conclusion that VR support in Unity is a complete mess that’s shoehorned into the different rendering and post-processing pipelines in a multitude of awkward ways. That makes it hard to know where to even begin with integrating FSR2.

I think the best way forward would be to generalize the texture array input/output support that I already added for HDRP some time ago, and then leaving it to each render pipeline integration to manage which buffers and array indices get passed to FSR2 and when. That means the core FSR2 dispatching code won’t really have to know about VR, it just gets some buffers that it needs to upscale. Any other approach would massively over-complicate the core FSR2 classes and make it that much harder to maintain.

Anyway, I guess what I’m trying to say is that VR support is not a trivial thing; it will require a bunch more research and each render pipeline and stereoscopic rendering variant will have to be considered separately.

Agree re Unity VR. My Fractal raymarcher game began around 5 years ago… I’ve honestly forgotten what version of unity I started on, but the OG VR support the raymarcher worked well on oculus1. Every version of unity since broke my raymarch shader in some new way. I eventually gave up trying to keep it VR functional. I’ve spent months of effort trying to figure out what they busted until it just wasn’t worth it to do again as several times I’ve modified the shader to fix it just broke again in a new way next unity version.

[Astfgl]

@blaher

This is awesome… though motion is rough on moving plants. I read the section about motion vectors… any links or info on how to go about “rendering motion vectors” or what you mean by this? is this something in hdrp only? any way to fix blurry motion/ghosted mess of plants w built in render pipe?

Yeah, motion vectors are essential for FSR2 to do its job properly, and sadly there are a lot of rendering packages out there that don’t include motion vector support.

For Isonzo we’re using The Vegetation Engine, which also doesn’t support motion vectors out of the box. My first temporary solution was to disable wind effects on vegetation, which did the trick but obviously the artists were not happy about the completely dead and static foliage. We had to go in and manually add a motion vector pass to TVE’s shaders, based on Unity’s standard motion vector shader, adapted to use TVE’s vertex animation and alpha cutoff logic. With that in place our foliage now looks nice and crisp in motion with FSR2 enabled.

Unfortunately this is not something I can provide a generic solution for. It depends entirely on what vegetation system you’re using in your project and which features you do or don’t use. Best thing you can do is badger whoever built your foliage renderer to include official motion vector support. Otherwise your only option is to manually add a motion vector pass like we did. This is not restricted to any of Unity’s render pipelines, by the way.

[khos]

Well done! Sounds like a very useful asset/script. Silly question, on the github page, where do I do to download it, if I get FSR2Unity-master.zip it does not contain all the files, maybe I am doing something wrong.

[Astfgl]

That’s odd. The Download ZIP function is managed by GitHub and it should automatically include everything that’s in the latest commit on the master branch. I can’t really fix anything about that. Which files do you think you are missing?

Anyway, I would recommend you use a Git client to clone the repository instead of downloading a ZIP. That makes it easier to pull in any changes in the future. Alternatively, if you’re only interested in the pre-built post-processing stack package, then you should download that from the Releases page on GitHub.

[dklassic]

Hi there, awesome job here! I was wondering if you’re interested in creating an equivalent for DLSSUnity? Since DLSS Unity Module already exists only that HDRP is the only RP that has implementation using it (and actually publicly viewable).

I’m aware of the existence of DLSS - Upscaling for Unity. However, their package for BIRP which utilizes the same approach as your custom Post Processing package doesn’t seem to work on my project. On the other hand, your FSR2 PP package doesn’t work on my project as well, while your Component version of FSR2Unity works perfectly.

So I was wondering if you have any interest to implement a DLSS Component version? I suppose it is technically possible for me to try to bang my head against the wall for several months to adapt your work to work with DLSS, but it would be great if you’re willing to use your existing knowledge to do a (potentially quick) adaptation!

[Astfgl]

Hi,

I don’t have any immediate plans to work on DLSS, and I’m rather tied up in other projects at the moment. I could have a look at it sometime, but I can’t make any promises.

What’s an interesting question to me is why the custom PPV2 packages do not work for you? Are you already using PPV2 in your project? If so, do you have any modifications of your own made to it that makes it impossible to use these custom packages? Or are you using a different post-processing system altogether?

For Isonzo, we were using PPV2 before but were using an older version with a bunch of modifications made to it, so I couldn’t just use the custom FSR2 package as-is either. Instead, I ported the changes made to add FSR2 support over to Isonzo’s custom PPV2 code. The changes made to the PPV2 package aren’t huge (a couple of extra files, plus some integration code in PostProcessLayer.cs), so manually porting over those modifications is very doable.

If you insist on using the Image Effect component, it’s probably easier to adapt the component to work with Alterego’s DLSS plugin, rather than trying to build a new DLSS integration to work with the existing component. Either way, be aware that the Image Effect component isn’t great and doesn’t work well with any kind of post-processing. It exists mostly for reference, as a basic example of how to make an FSR2 integration.

[dklassic]

Thanks for replying!

I know why your implementation doesn’t work on my project, it’s because I’m using a dual camera setup which is out of the scope of your project (I’m the guy who opened this issue).
Alterego Games claimed that their implementation works on multiple camera (only that in BIRP only one camera can actually use FSR2, the other has to rely on other AA method, which makes sense), but actually running their PPV2 on my project ended up providing identical errors when attempting to use your PPV2 implementation.

I’ve known that I can slightly misalign Rect setting of cameras to prevent Unity from syncing the two cameras, so I can just perform reconstruction on the first camera which is much GPU intensive than the other anyways. So at the very least if a working Image Effect component version of DLSS exists, I can suppose just integrate it the same I did using your FSR2 component.

I’m just a hobbyist learning to make a random game, so I’m not really sure if I’m ready to dive into the nitty gritty part of rendering yet. But, well, I suppose when in the future I have no other stuffs to do, I’ll see if I can adapt your work or Alterego’s work to make DLSS run on Image Effect component, as a learning opportunity.

Cheers, thanks again for replying!