I’ve been working for many months now on some big game worlds, trying to figure out the best tools and workflow for large maps. I had assumed this was pretty well established by now and have been surprised at how much I’ve had to puzzle through to figure out best procedures. So I’m starting this thread to share notes with others grappling with the same challenge and hopefully we can help each other figure out the best way to do this. (Note: I understand that Unity 2018.3 will have some significant improvements to terrain, though they’re a bit vague at this point and I am not too confident that they’ll be ready for production in 2018.3, and we can’t wait for 2019.1.)
Here’s what I’m doing:
Terrain
I’m making a (new version of a) game set in Yellowstone National. Park; we’ll have three areas represented in the game, each 7x7km in extent. We use World Composer to pull down the map data, then refine the heightmap with World Creator, and then use Terrain Composer 2 to paint splats and lay down grass, while also splitting it into an 8x8 multi-terrain array.) Because these are realworld locations, we can’t design the maps for optimal performance – most places on the maps have very long (10-30+ km) sightlines, so we can’t use clever methods to limit the player’s views. Then we use MicroSplat because it offers incredible features and performance.
Streaming Setup
Then I generated low-rez LOD meshes for all the terrain tiles and set everything up with World Streamer to load 9 Unity terrain tiles around player, and then load the low-rez terrain LOD meshes for all other grid cells. I’ve only done a little bit of performance testing with that, but even in a build, I’m finding that FPS drops to 5-10 for several very noticeable seconds each time stuff loads/unloads. I hope/assume there are ways to optimize that, but haven’t looked into it yet.
Vegetation
I want to use VS for trees for obvious reasons, and particularly because of the awesome integration with MicroSplat (especially for distant tree shadows, the lack of which has been a big concern of mine). However, since my maps have long sightlines, I need trees on the mesh LOD tiles as well as on the Unity terrains. Lennart has suggested baking the trees to Persistent Storage and use the VS Mesh terrain component – even though that’ll mean 64 VS packages, I guess that’s what I gotta do, but I haven’t gotten a chance to try it out yet. I also haven’t tried the new multi-terrain options in the latest version of VS, but I plan on using those.
I develop on Mac and found that VS’s grass rendering is very low performance there (though Lennart is going to look at that after he gets a Mac), so now I’m using straight-up Advanced Terrain Grass for grass (instead of ATG shaders in VS – the ATG shaders are too gorgeous not to use).
I’ve been hoping that TerraFirma would come along and solve a lot of my problems, but Lennart says that it is still in development and release is not imminent.
So even after all this work, I feel like I’m still nowhere near having a final workflow/setup for high-performance scenes with these big maps. I’d be enormously grateful for tips from anyone who’s trying to do the same kind of thing, like @sharkapps@jeromeWork@recon0303
There’s a huge need for a thread like this. A lot of the available information is asset specific. There’s very little that joins all the parts of the process together, how different assets work best together, or that really discuss the detail of getting an optimised open world scene.
Before I chime in with my own findings, @gecko can I ask how do you generate your low-rez LOD meshes from the Unity terrains? Do you use an asset like Terrain to Mesh? Terrain To Mesh | Terrain | Unity Asset Store
And how do you instantiate them in the scene, especially while working hand in hand with WorldStreamer? Is there a way to tell WorldStreamer to load a specific layer outside of the direct vicinity of the player?
I use a custom script to batch convert the terrains to meshes – I’d share it but it’s based on a script in RTP (we just added some batch options). In World Streamer, I set up ring streaming so it loads Unity terrains nearby the player and mesh terrains farther away.
So currently my own tests have been based on using the following tools: Gaia for terrain creation, Vegetation Studio for trees, grass and details, MicroSplat for terrain textures (I’ve also tried CTS but like MicroSplat’s functionality and I’ve been led to believe that it’s better optimised) World Streamer for streaming terrain tiles and GameObjects
I’ve been reading that baked occlusion culling is not a good idea for large open worlds (maybe someone can confirm?) - I’ve read elsewhere that the Culling Group API for trees and bushes can be used to improve performance, but I know very little about it (Unity - Manual: CullingGroup API)
For splitting the terrain I’m using the free script recommended in the WS tutorial: Kostiantyn Dvornik :: Ideas: Unity split terrain script
To test things out I’ve built three versions of the same test terrain scene.
Build1 is a 2K terrain with Vegetation Studio doing trees and grass. Terrain was originally created with Gaia and there are a few farms and additional trees/rock meshes dotted about (from the initial Gaia stamps). Everything’s set to Static. Just a single scene no streaming.
I’m using this as my worse-case, baseline scenario.
Build 2, I’ve split the terrain into 64 256x256 tiles. I’ve got a Vegetation Studio System component on each of those tiles, each with its own baked Persistent Storage asset. Farms/rocks etc I’ve set to their own Large / Medium / Small layers for World Streamer.
Build 3, uses the same tiles and World Streamer set-up. The main difference is that each terrain doesn’t have its own Vegetation Studio System. Instead, the main game scene has a script that creates a pool of 12 Vegetation Studio Systems and allocates them to terrains as they are loaded (the baked Persistent Packages for each terrain are also added at the same time)
this appears to be the recommend method from the advice the Lennart and others are giving on the Awesome Technologies discord (Awesome Technologies). * I’ve @OneManBandGames to thank for his advice on this methodolgy, and for sharing his VS System pooling script. (I’ll check with him whether he minds this being shared on this forum)
The results I’m getting aren’t quite what I was expecting, and they’ve just been confirmed by @angelekas80 on the Discord.
Build 1 has the lowest Mem allocation: 581MB
average 23FPS (low but smooth and steady, no spikes) on low end PC/GPU [windows10 xeon E5-2630 / 4GB QuadroM2000 / 16GB RAM]
42FPS on higher end [windows10 i7 6700k 4.00hz asus gtx 1060 6gb oc 16gb ram ssd samsung 500gb]
Build 2, has a slightly higher mem allocation of 541MB,
higher 27FPS on low end
similar 40ish FPS on higher end PC
both have lags (frame drops to 14fps) as different parts of the terrain load and unload
I’m guessing this is a World Streamer issue which @gecko has confirmed he’s having too… Just to clarify, I am testing the build not in the Editor where I know async additive scene loads don’t properly work )
Build 3 has the highest Mem Allocation at 660MB
27FPS on low end PC
40FPS on higher end
and really bad freezes and frame drops on loading and unloading tiles. (again partly a World Streamer issue I expect, but maybe VS is responsible too as each System initialises from the pool?)
So. TLDR. There doesn’t seem to be much gained by splitting the terrain. (mem allocation increases and hiccups are introduced as WorldStreamer and the VS pool load and unload) Not at all what I was expecting!
I’ll keep testing and trying things out, and reporting back here, but would be incredibly grateful for any advice or guidance.
Builds can be got from here if it’s of interest: VS-WorldStreamer_Tests1.zip - Google Drive (2GB zip) Using Unity 2017.4.2 and not likely to move on to 2018 for a while, for stability and compatibility reasons
BTW, I’ve also wondered about loading all 64 Unity terrain tiles, but then disabling the terrain mesh on distant terrains and using mesh terrains there, but still drawing only the billboarded trees…since I think the latter are pretty optimized, and it’s the terrain itself that is expensive. But then I realized that Unity would still have to calculate the heightmap in order to draw the tree billboards (at least, I suspect maybe it does, but what do I know?), so I haven’t tried that yet. It’s very hard to find definitive information about the terrain system in that regard.
Reporting back on some research re. optimising Unity terrains to mesh:
“export the terrain as an OBJ and open it in zbrush. From there you can “decimate” the terrain down… After removing the triangles you can then apply a normal to it and it looks absolutely perfect. So on a mobile game, I have a 2kx2k fully detailed terrain at 5.6k tris, 7.9k vert and at only one draw.”
Interesting idea, although a reply mentions the following downside:
“For the texturing aspect it might be a bit limited unless you find a splat map shader for the terrain. Otherwise, using a 2k texture map for the terrain would mean every 3 feet give or take would be covered by 1 pixel! That’s a pretty blurry terrain. It doesn’t improve all that much using a 4k texture map either. I suppose using tiling textures with a splat map shader is the best way to go.”
From the same thread, this looks like an article that’s full of really detailed information:
Summarised by the author:
"I can compare the performance on iPhone 4 to iPhone 6 for unity terrain and mesh terrain. In the game two Unity terrains of 1024x1024 datapoints with 3 splatmaps were shown with adapting their pixelerror during runtime every second depending on the distance. The mesh conversion splits the terrain in square patches (T4M unfortunately does long stripes), so only few of them are in the frustum. Then the mesh topology was simplified to a few percent of the original by an asset called Cruncher at development time, so that the look in the game is approximately the same.
At runtime the Unity terrain takes a lot of CPU performance and batches, pops visibly from time to time and is 20% slower than the mesh! Anyway we shipped it with Unity terrain because it still looks nicer when you look closely. If one spend more time to regulate the crunch process better, the mesh is the better option. In your game it also depends if the valid camera position has a defined limited area or the player can cruise around freely and if your app is already CPU bound or not."
I’m reading a lot about Atlasing being a key part of terrain optimisation (I knew that for ‘normal’ geometry) but hadn’t really considered it for terrains.
what would the process be for atlasing vegetation like SpeedTrees or TurboScalpeur grasses dropped into Vegetation Studio? (could something like MeshBaker actually be used?)
Next up, I’m planning to create those low rez terrain meshes an implement ring streaming. I’ve just found Mesh Terrain Editor Free which hopefully should do what I need.
Thanks for creating this thread. I agree with @jeromeWork that we need a place for the most accurate and up-to-date information on how to put together a reasonable workflow for building large open worlds in Unity. I’m happy to share whatever I have learned in the many months I have spent working on one of my most fun yet frustrating projects in Unity so far.
I love what you have done so far, @gecko , with your WolfQuest 3 work! Yellowstone is an inspiring environment to recreate and you’re doing a good job with it from what I’ve seen.
The biggest thing I have learned (and honestly expected at the outset) is that there is a lot to learn, and there are a ton of things to consider when building a large streaming open world game vs. “small world” games. Just getting textured terrains working smoothly without hiccups in ring-streaming is a challenge in and of itself. It’s really important to first get blank terrains streaming without spiking. In some versions of Unity, this appears to be impossible due to bugs (see below). If you can, however, get the stars to align, smooth streaming of terrains is possible.
My Project
My project is really just a prototype at the moment that will allow the player to roam the open world by foot or by various vehicles (cars, trucks, boats, planes, etc.), as they try to accomplish tasks. I’m developing the prototype as part of my self-driven education in game development (all of my professional experience is software development and testing, but mainly for servers & back-end services).
Terrain
I am also using TerrainComposer2, but am not using WorldComposer for my this project. I have tried to use MicroSplat in this project a couple of times, but I ran into some issues when I needed to go back and tweak things on the terrain splatmap in TC2, so I removed it and I decided to try it again later when I really need it.
My terrain is composed of 400 (20x20) 512mx512m terrains. The setting is a tropical island, so I am using the Tropical Forest Pack for the foliage.
As you can see from the above video, I’ve got the LOD0 terrain streamer working smoothly.
Right now, my challenge is to get the scene with both LOD0 and LOD1 terrain streamers running smoothly. At the moment, I am having issues with garbage collection.
You can easily see in the video where the spikes occur. I attached the profiler to a Development Build of the project, which points to garbage collection as the culprit.
This image shows a snapshot of the profiler connected to a 64-bit Development Build running on Windows 10
Right now, I’m guessing that the unused assets getting collected here are unloaded LOD1 mesh scenes. I am not sure how to correct this behavior other than by not ever unloading the LOD1 scenes, but then my concern would be the the amount of memory needed and also why bother even streaming LOD1 terrains at all at that point?
I’m hoping someone here has dealt with this before and can offer some advice.
The Future
I know a new terrain system is coming, and when I was playing around with RoadArchitect on 2018.2.0b11 the other day, I noticed the terrain looked different and behaved differently. I am curious what is changing and how it may help out for open worlds and/or streaming. I hope to migrate a copy of my project over to 2018.3.x soon to test things out.
Unity permits developers to control the priority of background threads that are being used to load data. This is particularly important when trying to stream AssetBundles onto disk in the background. The priority for the main thread and graphics thread are both ThreadPriority.Normal – any threads with higher priority preempt the main/graphics threads and cause framerate hiccups, whereas threads with lower priority do not. If threads have an equivalent priority to the main thread, the CPU attempts to give equal time to the threads, which generally results in framerate stuttering if multiple background threads are performing heavy operations, such as AssetBundle decompression. Currently, this priority can be controlled in three places. First, the default priority for Asset loading calls, such as Resources.LoadAsync and AssetBundle.LoadAssetAsync, is taken from the Application.backgroundLoadingPriority setting. As documented, this call also limits the amount of time that the main thread spends integrating Assets (NOTE: Most types of Unity Assets must be “integrated” onto the Main thread. During integration, the Asset initialization is finalized and certain thread-safe operations are performed. This includes scripting callback invocations, such as Awake callbacks. See the “Resource Management” guide for further details.), in order to limit the impact of Asset loading on frame time.
How did Campo Santo get smooth performance in Firewatch?
This video is referenced by people in the forums all the time when people ask about getting good performance in open worlds. It’s a good talk and it has a lot of great tips for achieving great performance in Unity, but ultimately they revealed they needed to change code in the Unity source to get certain things to load over multiple frames.
I wonder how much can be optimize by building our own terrain directly. I mean a terrain is just a grid of vertices with only Y coordinate changing, so we can reuse the same data and only update the Y coordinate (the heightmap, only an array of float), LOD are just bigger terrain of the same size, which mean reduce density. Reusing the same data would maybe lessen the streaming hassle. And since we would have our own terrain, we can organize the heightmap streaming more efficiently by breaking the data in neat friendly package we can reassign or progressively read as we need, instead of trying to go around unity limitation. I was considering using similarity matrix to find recurring objects to stream from neighbor terrain and thus limit the data needed by only storing taking the delta difference per terrain, so we don’t reload existing object but only instance new version of them using the delta. And since we could precompute that difference, have a script that warn when we have over budget differences.
I have been researching that a lot, and that’s a common occurrence in every single one of them. It’s either change the source code, and devise another way to implement similar thing than unity.
Thanks @sharkapps That’s a stellar contribution! A lot there to read up on and digest.
I did do some testing with low-rez LOD1 mesh terrains and similarly found very little in terms of performance gain - but as my project is more of a walking sim, with plenty of coverage stopping long sight lines, I quickly realised that the long distance ring streaming just wasn’t necessary for me. It added an unnecessary overhead, which unsurprisingly slowed things down.
Can you clarify how you’re adding your foliage? Are these trees meshes with LODs, or terrain trees? Are you using something like Vegetation Studio? Or just placing them on their own on World Streamer streaming layer?
Agreed that baked occlusion culling isn’t a good idea for a large world. Have you implemented any other solutions? On that tip…
Another tool I’ve yet to try, but intend to, is InstantOC which I’ve used very successfully in the past on a procedural dungeon crawler.
Vegetation Studio is something @gecko and I have in common on our separate projects, as well as MicroSplat. [On the WorldStreamer thread]( [RELEASED] World Streamer ! Forget about your game memory usage and create big world! page-29) he’s understandably being told to strip everything down to bare terrain and LOD1 meshes to see if the hiccups persist. Which is very much your advice, and totally makes sense.
Someone called Timotee posted some incredibly useful info on the discord, outlining some of the initialisation issues with VS. I hope he won’t mind me reproducing it here, for posterity:
“We are using SECTR and doing ring streaming. Before Vegetation Studio integration we had minor slow down upon loading of each terrain sector, I’ve really worked hard to mitigate and load anything that comes in over time and not all in a single frame. We are using 1 Vegetation System as a child object of each terrain and only using persistent storage on each. After some profiling in exe, I’ve found the OnEnable of the Vegetation System to cause some pretty bad hitching with this process. Looking at the code it looks like VS uses the currently assigned terrain bounds to allocate memory and create vegetation cells in OnEnable. I think ideally for streaming like this it would be best as 1 Vegetation System for the whole world, this way your allocation hit is all up front upon loading into your open world scene. Then just manually assign an arbitrary bounding area set to encompass the world for vegetation cells, instead of using the currently assigned terrain OnEnable. For us since we only use persistent storage we don’t need to worry about run time spawning, I don’t believe the actual terrain angles/heights and splat data actually matter to this, so the assigned terrain is really just a bounding box. Now if you have run time spawning VS would need to know as each of those terrains is loaded and run time spawn on each load. It looks as though VS Pro does this type of optimization, 1 Vegetation System, arbitrary bounding area and loads as terrains come in. I just don’t think we will have time to wait for VS Pro, we need this optimization before then.”
Inline with @neoshaman I’ve been trying to think a little out of the box and spent the day trying out MapMagic. I haven’t got as far as adding vegetation yet but it does look very promising. Very smooth generation of an infinite terrain. Of no use for @gecko 's Yellowstone recreation but may well be a potential alternative for me. As long as I can add handcrafted details. Sadly my main workstation is in for repairs so difficult for me to do any realistic test, but should hopefully be back up and running properly in a week or so.
Right now I am just adding trees, grass and rocks via TerrainComposer2. The trees are meshes with LODs. I was experimenting with Vegetation Studio in this project a while ago, and I had a working system that would generate the necessary VS components as each LOD0 terrain loaded, but at the time there wasn’t a good solution for dealing with the LOD0 terrain meshes in the ring streamer. @LennartJohansen advised that the VS Pro update he is working on will make setting up VS with streaming multi-tile terrains much easier and much faster, but it will require Unity 2018. The quote you included from Timotee is a good summary of why the current VS isn’t good for streaming. Given all of that, I have decided to wait for VS Pro before I attempt using it again in this project.
Can I be a pain and ask one of you guys with a half decent machine to test this build out?
It’s MapMagic (set to infinite repeat tiling) Vegetation Studio with a decent amount of grass (2 types) and 1xSpeedTree, CTS for terrain texture, plus Enviro, Aura, and a bunch of post-processing which should make it crawl to a halt but it actually plays at my end - not great but I’m surprised it even runs. MapMagic_test3_CTS-VS.zip - Google Drive (300MB zip)
I think you’re right, it’s all on the GPU. GT705 v GTX750ti is the main difference
Agreed that MapMagic procedural generation is different to streaming, but appears to be essentially doing the same thing - tiling terrains and loading/unloading as the player moves through the grid. There’s obviously something it’s doing that’s a lot more efficient than WS?
I’ve watched again the Unite Berlin video (at 32:50) about terrain improvements coming in 2018.3, and seeing how 2018.2 just released, I’m more hopeful that 2018.3 will be released earlier (Sept/Oct) than later (Dec). Apparently these improvements boost loading performance tremendously, and also possibly eliminate the need for mesh LOD terrains.
So I’m thinking maybe I should just set this aside for a month and see what the 2018.3 beta can do then. No sense pouring a ton of time into this now if that’ll solve many of my problems in the not-too-distant future.
Only problem is that 2018 brings with it a few more idiosyncrasies that may introduce new bugs or make using certain other assets problematic. Personally not sure if I’d trust it for a proper release…
you bypass a whole host of problem with that, like non continuous data.