My feedback on Virtual Texturing.

I have been using VT on 2021.1f9 HDRP11 on an open world project for the last year. I have been using VT since it came out (on and off) and now my project fully uses it all the time.

I would like to provide a detailed feedback of my experience and hopefully be able to share important notes about the new tech.

So why do I need VT?
Well my project is an open world game, innately having towns in an open environment. This means that even with careful level placement and occlusion, some angles, parts of the map will present more texture than others. For example, a town. So, working with 4k~8k textures, and given that HDRP uses 3 maps on standard, reaching 50+ 4k maps was very easy with few props and walls placed. This, essentially floods the GPU pipeline and causes a massive lag until the texture is uploaded.

Isn’t texture streaming enough?
Yes, and no. While texture streaming improves the lag situation and the saturated GPU pipe, VRAM use starts to go up out of control and this is for a small town with not much props.

So, then, VT.
Yes, VT, solves all this issue, and to my surprise, it just works and this is not a deal you get everyday in the Unity eco system. I can easily run 4k~8k textures upwards of 100+ maps as long as they are well placed.

Is the VT ready for production?
Yes and no. Yes, in the sense of stability. It is stable, and works as expected. And like I said above, this is big.
And No, in the sense that it lacks some basic features that should have been included from the beginning.

What exactly is missing?
I am not going to mention QOL improvements as they are just conveniences. But the issue that I am about to share is critical and renders the tech “not ready”. In my opinion, there is only one real issue with VT at the moment, one which needs to be addressed immediately.

So what is it?
When VT is loading in, instead of showing the lowest level mip map (like every other VT solution), Unity’s VT just shows black color until it loads in. And this is what it looks like.


This pop in from black color → texture is very noticeable as it mostly happens close up. It mostly occurs when entering/exiting a room and unless there is a one way type of corridor, this is hard to workaround. This probably will make my players very very unhappy - no doubt.

So, devs, please fix this asap.

Here is a visualization of what I think it should look like. Instead of black color, lowest level mip map is loaded in first → then vt texture. I am think maybe at shader level, lowest mip map can be shown (not as vt) if vt texture is not ready.

4 Likes

One more thing, if you guys can add a bool of sort for shader graph (eg. if streaming not done) then we could write the behavior ourselves. I am assuming that this is not a lot of work as since the shader itself is capable of displaying black color while the texture is not ready anyway.

Hope this helps.

Thank you very much for your detailed feedback!
The VT system indeed does not do any automated prefetching at the moment for objects that are not visible. The black rendering means that there is no data for that uv space in the cache. I think its a good idea for us to provide this. Currently, you can create a prefetcher yourself using the RequestRegion api. Did you try using that?

You can find an example script in the VT forum thread [here]( https://discussions.unity.com/t/781264 page-3#post-6444272). A sensible approach seems to be to request the top mip (lowest resolution, which is the mipmap that equals the tile size) for each material that is within a certain distance from the camera. This way, there should already be some data in the cache once you enter a room. An alternative is to just load all top mips although this can use a lot of memory if you have many textures using VT.

1 Like

@AljoshaD

Would you also please consider an option to stream lightmaps and shadow masks like VT textures?
I can do it with lightmaps even now by using them as regular textures instead of the Baked GI Node. But this approach seems pretty ridiculous because then I’ll have to create a material per each lightmap.
Also, in the light of the recent events in the industry, maybe it’s time to start streaming virtual shadow maps as well?
Thanks!

I think you are misunderstanding what I am trying to say. So let me try to explain again and hopefully I will do a better job this time :slight_smile:

So, what I am trying to say is that VT shows “black” when the texture is not available in cache and shows black until it becomes available.

Then you are saying call RequestRegion, which is basically spamming the VT cache to alert the important textures.

I am saying that this RequestRegion doesn’t work well unless we are talking one way (room with a one way corridor).
Another use case is warming up the initial position of the player camera so that the world is not full of black patches when the game starts.

I get this and I am saying that this is not enough or good enough for production.

What I am suggesting is to display the lowest mip map (lets say 64x64 and this is not from VT cache) while the cache is getting atlased in VT cache.

So basically, show non VT texture so that the pop in when the VT textures are ready is not too noticeable.

Now, if you can add a shader bool for shadergraph to check if texture is ready in VT, then we can use both VT and non VT texture (so, I won’t be able to mark VT only in texture import settings I assume) to swap out accordingly. The lowest mip map not VT texture will add some overhead, but it will be instantaneous and most likely will not slow down by much as they will be very very small in terms of size.

Or you can make it so that all VT shaders behave this way if you find that more appealing. Either way, I am saying that we need this feature one way or another. The RequestRegion just adds way too much work and starts to interfere with level design and such. Not good given that Unity already forces us to design levels in certain ways (light map, nav mesh etc.) another layer of level design enforced by VT is not welcomed.

I hope this clarifies what I was trying to say.

Can you elaborate why it does not work for production? The API should allow you to implement any kind of prefetching.

The VT system should load the tiles from low res mip maps first to have data in the cache sooner. If you only use the automated screen space resolving then the cache has no data for a few frames for newly visible materials. The approach you suggest sounds good. You should be able to script this with the current RequestRegion api, although at the minimum you need to load a 128x128 VT tile instead of having the small 64x64 regular texture that you are proposing.

Do you mean it adds work for artists/designers? To be clear, you don’t need to add a script to each object. One script in your scene is enough. It should be straightforward to add to a scene. Did you try the example script?

yes we’ll make sure to consider it for our roadmap

1 Like

I am aware that there only needs to be one script, but I assumed that the performance impact would be there like you mentioned. Your suggestion (nearby textures at 128p prefetched via VT) vs my suggestion (64p non VT textures working like normal unity texture) might yield same results with no performance difference. I will implement your suggestion in the next few days and get hard results and I will post back. I think it needs to be verified before I make further assumptions.

  • This is not production code and should only be seen as experimental.
  • This is not guarnateed to be the most performant implementation.

On the other hand, is there a reason why the lowest mip is 128x128? 64 vs 128 is not much and probably will be alright but just wanted to know if there are technical limitations.

yes, the smallest mipmap is 1x1 tile and the tile size is 128x128. We don’t store data that is smaller so you cannot sample a mipmap in VT that is smaller than the tile size.

Okay I get this error.

ArgumentException: The provided mipmap range is invalid (max mip: 9)
UnityEngine.Rendering.VirtualTexturing.Streaming.RequestRegion (UnityEngine.Material mat, System.Int32 stackNameId, UnityEngine.Rect r, System.Int32 mipMap, System.Int32 numMips) (at <4744a1d6377147c79a323f95fb07c941>:0)
VirtualTextureController.Update () (at Assets/_Scripts/VirtualTextureController.cs:87)

So I modified the code a bit to debug what is going on and it looks like Unity fails to get texture size(stackSize) for some textures, and hence getting ridiculous mip level values like 2147483641. I think your math is correct but some how Unity fails to fetch texture size randomly. I have looked at import/format settings for textures and could not find anything different from the ones that work vs that do not work.

I ignored some errors by not adding to the m_stacks if mip is less than 8, and now for the textures that return the correct mip values show correct textures but seems like it is rendered wrong. They appear green instead of black…

Also, I was thinking if we can just set a single color, metallic/gloss level per material instead of preset black color. It might just make things less obvious with very little effort. So, show a general color and material instead of meddling with mip textures, seeing that there is something wrong here. Unless, I am doing something wrong :slight_smile:

Also, while the script is not functioning in a normal state, the performance impact is really big. One, it created a huge dip every third frame, and also eats quite a few ms out. I can see this potentially eating away more than 3ms in scenes with many materials eg. open world. The only way that I can think of, while still using the request API, is to add some distance/zone aware logic which at the point makes VT a lot less appealing…which is why I said it limits level designs in some ways as zone trigger, spatial awareness needs to be put in as well.

One way I can think of is to store a list of materials and mip level integer as a scriptable object for zones and just feed it in in runtime. This is pretty much prebaking data and I am not sure I want to set the system up just to avoid black color pop…hence my suggestion to show lowest non VT texture while it loads in.

Any idea why it fails to fetch texture info?

Okay, I’ve spent the last few days tackling the problem with the RequestRegion API, and here are my thoughts.

  • RequestRegion is not the solution to the black color as no matter what I do, even loading the top mip fails to happen in time fast enough (even when I am spamming it every update). It simply fails to fetch the texture in time as the VT atlas does not resolve in time and it seems to have different results everytime. It works just fine in simple scenes, but in a big enough scene, for example a town in an open world environment, things just fail to work in time, randomly.
  • The RequestRegion API, introduces a ton of frame time fluctuations that are very noticeable, and this is with around 20 materials. So, in all honesty, this is not a method that is viable for 99% of the situation.
  • I made a zone system that will selectively spam RequestRegion and this fails as well. I am not 100% sure why, but my guess is that when switching zones and the materials to request, it need a lot more time to flush the atlas and add (even the top mip)
  • I am using 2 x NVME drives, both 2TB each, and is probably a much faster SSD than most people will likely have and depending on the SSD situation, the black color display time can sometimes go on for 10 seconds.

So, I think I have tried pretty much everyway to use RequestRegion API to mitigate this issue, and as it seems, it fails. The issue becomes more prominent as more textures are flushed in and out (ie, by going into different rooms with different textures) and as the scene scales up, the issue becomes a huge issue. I have had objects staying black for as long as 2 seconds. It is beyond a quick flicker of colors and looks like the renderer is broken. I am posting my cache settings in case you are interested, but as you can see my cache is much bigger than recommended.

Also, the RequestRegion API method introduces a strange behavior where some parts of the texture will incorrectly display the top mip map even when higher res textures are already being used on scene (look at the floor, there is a clear line that divides it and it is not a near screen blur but an artifact of VT)

So, I think all in all, we can conclude that the current implementation is not ready for a full blown scene and it breaks apart very quickly unless, you drop texture resolution which therefore will make VT atlasing a bit faster and hence a bit less noticeable, or by making important or immediate objects not use VT.

Conclusion?

We need a guaranteed way of showing textures not routed via VT (top mip map) until VT is ready to show something. A boolean check, or a shader implemented method should both work, but we need something. In current state, it does not work in proper scenes. It work well in a simple test scene though - it is very hard to notice. If this is not guaranteed, then this beautiful and elegant tech is NOT READY FOR PRODUCTION. There just is no way to display texture in time in a scalable scene. It is the most basic feature of VT and yet it is missing…

Another method I am working on, as I am almost certain Unity is not keen on improving VT immediately and dwell on it for a year or so, per usual, is to use LODs, whereby the furtheraway objects will simply display lowest mipMap texture (non VT) and will fade in to VT material. This way, VT will start to request texture but will not be shown until the camera is upclose. The issue, is that it needs to be time based or bool based (IsVTTextureReady for example) but will be distance based as it is LOD. I am not sure if this will work but I will have to try.

If devs can reply and suggest a different route, or work with me on this, it will be greatly appreciated.

Quick Update: I found out why RequestRegion was failing. It turns out StackID can only be obtained while the object is actually rendered (I am not sure if the StackID changes if the object is disabled and reenabled) but none the less, after I got it to function as intended. I was able to cut off a few frame costs by pre calculating mip levels and material and feeding it to region update as the camera traverses the world. I was not able to notice that VT was failing to feed mip maps as the error never pops up in the editor, only in the standalone build in dev mode.

As long as I keep persistent materials to below 10 and dynamic materials (that gets turned on and off) to about 10, so about 20 total, the performance impact is not so bad, but introduced a lot of hiccups to the frames.

So, the dev’s suggested solution works albeit the documentation fails to mention a few important things. It adds another layer to my work (It is definitely not just turn it on type of thing) and is near impossible for objects in open (unless I consistently spam the API, which costs performance, so no)

After a full week of trying VT on my commercial open world project, I have mixed feelings.

  1. The feature feels unpolished in the sense that non VT top mip map option is not available. Feels like a missing basic feature.
  2. Some crucial details regarding RequestRegion API is missing documentation and the provided example script can be misleading as it might make people think that it actually works (it only works if all materials are visible/rendered, otherwise it fails to obtain StackID → fails to feed to VT → this error is only shown in standalone dev mode)
  3. The RequestRegion API eats a bit of frame time but it cannot be split up over multiple frames as it needs to be spammed every Update, hence the more materials needed to be requested, more frame loss. This is a fundamental design issue where the cost of a custom function increases proportionally, the best would be a fixed performance cost.

This is my first VT experience so I may have some absurd expectations but my expectations were pretty much based on other games that had VT implemented and they never had black colors popping…whether they use the same RequestRegion type of API and performance impact, that I have no idea as I can’t tell how a feature works under the hood of a particular game.

Nevertheless, I hope non-VT mip maping can be implemented to avoid all this fuss. I can’t really see why it would be difficult to do this in shaders, two texture sets blending if a condition is met, so I hope the devs can look into this. People are going to talk about this as more people start using it anyway. So, it might be good to have the feature ready before it is requested widely (it has by me, but it seems like VT adoption is not happening much at the moment). I think I will end my venture with VT here.

I hope my post and findings will become useful to someone in the future.

Peace.

1 Like

Thanks again for your detailed feedback!

With regards to the green color that you are seeing, that seems to be a bug. Would it be possible to submit a bug report with a limited repro case?

This seems like a good work around but we are focused on improving the VT system so that you don’t have any black rendering or latency issues. Ideally, you don’t need a copy of the texture in a regular format and only have the data in a VT format.
The performance of the RequestRegion API is indeed not very good and that needs to improve significantly. Also, the fact that the stacks are not created when nothing is rendered should change as well so prefetching can happen before rendering.

Thanks for sharing, this is very useful. Although the sizes are not low, you might indeed not have enough memory to prefetch a lot of data. If many of your materials use one specific texture format, lets say BC7, then 265MB is not high, especially if your framebuffer has a high resolution. We have an automatic quality reduction mechanism but this might not work well with a lot of prefetching, we’ll investigate.

Thanks again for sharing your results after trying the Streaming Virtual Texturing feature. The feature is still in preview and we still have some work to do. Your feedback helps us to focus on the most important areas.

Thanks for the replies, and honestly I feel like the core of the VT is quite solid, maybe except for the part where how VT chooses what needs to be streamed in as a priority (I am guessing how much of the screen space it takes up, just guessing from how it behaves). The non VT option, I believe is a must in my opinion but I also get the feeling that this is not the direction you guys want to go to so I will not mention that anymore. But could you at least allow us to set the color (instead of black) at shader level so that the pop in is not too bad?

Even with RegionRequest API, in an open environment, it is very hard to make the pop in be not noticeable. For example, a box in the middle of nowhere visible from long distances, it just can’t be handled well.

Option 1) Spam RegionRequest → Not good performance wise, and it needs to be spammed from quite a far, which just adds to the issue. Also, often times, VT will elect to stream it late as it will be small from some distance.

Option 2) Have LOD with lowest object a non VT shader and close up, VT shader. Same issue where the standalone has to have 2 texture sets and if not done right, LOD transition will show that black color during blending. Also, this means I have to design each placement of object per location to make the transition look good.

But all this issue can be fixed or minimized by simply allowing us to select a color, by then we can just make it less noticeable. Easy and simple.

that is certainly doable. The color would be the same for all textures though with the current implementation.

Color would have to be per shader. I am guessing the shader doesn’t know whether VT textures are ready or not, right? So, maybe add a shader bool so that we can check if texture is ready or not.