TerraLand 3 - Streaming Huge Real-World & Custom Terrains For Open-World Environments

This is a good question to be brought up here. Actually never tried WorldStreamer and don’t know how it works but I can guess that it divides terrains into grids of tiles in the editor and then loads/unloads them in realtime! Is that true? Maybe the developer can come up here and explain the process.

TerraLand’s Streaming system on the other hand, creates data tiles (heightmaps, satellite images, splatmaps, normal maps & normal textures or colormaps) and put them in a cached server whether in a local hard drive location or an online destination and then loads those tiles from these servers in real-time and creates terrains out of them from scratch.

As it does all that outside Unity and terrains objects doesn’t have to be in the Editor, we are not bound to Unity application’s memory and drawcall limitations which is existent in a static scene but as it loads data on demand from an external source so the capacity of the data and resolution can be possibly endless and be terabytes in size.

For the rest of the process you mentioned, it seems that it is already a neat solution to stream all assets, but I’ll comment later when you give more info or show any media if possible. Regarding terrains only, take a look at the following video which renders a 160 x 160 km area with the total resolution of 16384 pixels for the heightmap and 65536 pixels for the imagery out of a cached server with 1.5 GB of data in size.

https://www.youtube.com/watch?v=jOkJOjjmCxE

Also, TerraLand is not limited to real-world terrains and any custom made terrain object or data can be streamed in runtime. Following video shows a high resolution terrain streaming in runtime out of World Machine renders placed in a cached server on a local hard drive location:

https://www.youtube.com/watch?v=__kVk7uBMDA

Video tutorials for the 2 above videos are also available on our YouTube channel.

1 Like

Thanks for the reply, I appreciate it and will definitely watch the videos. I was talking with someone else via email a little while back and they asked how WorldStreamer works as well and this is what I had wrote to them below, I apologize it is a little long and there may be a few sentenses that may not pertain exactly, but I believe I got a pretty solid bit of the details in here.

My main concern with whatever I used ends up being the fact that my game is an online persistent world RPG, so I don’t think / know if floating point fixes would really work for my game, as things can’t really just be moved around willy nilly like most fixes do. I am trying to make my game as modular as I can, terrain wise as well, so I can just start out with X size and then expand on it as I discover ways to keep the world seamlessly loading but also not have to worry about the distance from 0,0,0, but that might end up having to come from some sort of multiserver solution at some point.


"With WorldStreamer you are to have to terrain pre-sliced and you put in size of each slice that you already have. What it actually does create your different streaming layers. I use WorldCreater stand alone for my terrain which has a sync tool that brings the terrain in from WorldCreator and automatically slices it for you when it imports, but of course not everyone has that. My map is 10248x10248 currently cut into 1600 256x256 chunks of terrain. Yes, a lot, lol, but It gives me a lot of flexibility in how much or how little I want to load because the loading grid system is dynamic. So for instance If I get on my drag and fly upward I can have it trigger to extend the grids loading range from 3x3 to 4x4 or 5x5, etc and vice versa, when I land it goes back to 3x3. So within the SceneSplitter, I put in that my terrain size is 256x256, then I create my layers. What you do is prefix all of your items in game with something (which is why he mentions another free tool which can auto rename with pre_ or _suffix based on different things).

Each prefix becomes a layer. So all my terrain Ter_, so it is Ter_WC_Terrain_01x01x01 or whatever, so that takes care of my terrain layer (for performance purposes terrain is always by itself when loading or unloading) My next layer is my mesh lod which is LOD_Ter_WC_etc, then for the time being I had My whole castle as another layer by itself just because it was the only thing really else that I had in the game at the time name Castle_Castle. So that would be 3. The terrain layer gets set more or less as the primary, it will always load that first when the game is loading, hopefully before you player loads, lol. It has a Neighboring_terrain setting/script on it which I believe is used as the connecting point for the grid system for loading and what not, but I am not 100% sure, never really looked through that file. When you tell the scene splitter to do its thing, it takes each layer and all the objects within it and then based on each terrain chunk exports every 256x256 area as its own scene. So each layer would end up with a folder based on the layer name and in there ends up being one scene for every 256x256 area but only containing what was in that particular layer based on its name, and which scene it is in is of course based on its location and which chunk it was in. Along with that was a single prefab with a script on which contained a list of all of the scenes within that layer and those prefabs are what you put in the scene and attach to the rest of the components to make the system work. The way it named each scene dictated its position in world space as seen here. 2018-04-25 - 19.25.27 ~ Cropped Capture - Imgur This allowed everything within a particular chunk of terrain to be able to be loaded in the right place, but at different times.

In your grid loading settings, you would say I want the terrain layer to load only 1x1 grid space from me, which would be the one I am on, and then the first set of grid spaces around me but nothing further than that. The next layer is my LOD mesh terrain, using the Ring LOD system it works by setting a minimum and maximum loading range. So I have it set to I want it to only load in grid spaces 2-3, so that terrain is the first grid space, but then 2nd and 3rd grid spaces are the LOD. If you have more layers, say a large and small object layer, you would set it that you want LargeObject_streaminglayer_ to begin loading as far away as 3x3 grid spaces, but you only want SmallObject_streaminglayer_ to load then you are 1x1 grid spaces away. Even though one particular grid space within the world might be loaded, it might have 4-5 different scenes of the same area, each containing different objects depending on its streaming layer.

It works really well as long as you take the time to prepare all your assets by naming them in the hierarchy properly. The only downside other than the extra work is having extra size on the client because of all the extra scenes, even though each one might only have a few objects in there by themselves, each scene file takes up space. Well worth it for the performance gain you get though. Mix that with the trigger loading system and you can have some huge open areas that look populated (VegetationStudio will let you prebake billboard foliage and attach it to each chunk of the lod, but use the real stuff on terrain when you are close to it) but then unload a bunch of that when you enter an enclosed area with the trigger and have super detailed interiors.

The way it tries to combat too much loading at a single time if you cross the threshold into a new grid space is simply allowing you to input a set number of frames for each thing to wait in between its loading, so you have terrain load first frame, wait 2 frames then something else, wait another frame for something else."

1 Like

@MostHated thank you so much for the explanation about the other asset.

Regarding terrains only, the whole concept and behavior seems to be close to each other except TerraLand loads terrain data from outside Unity and creates terrain objects during runtime as I mentioned earlier.

This has its pros and cons though;

  • The advantage is that the data can be huge and you are not limited to memory as nothing needs to be setup in the editor. This makes it even simpler as no manual setup is needed and everything is automated and handled by the “RuntimeOffline” script.
  • The disadvantage is that it needs to process data at runtime and create terrain tiles which affects performance instead of simple loading/unloading of tile objects, however heavy processes are under multi threading functions if possible or carried out between coroutine calls.

The reason behind this method we use in TerraLand is because we focused on streaming huge datasets as our customers are mostly involved in bigger GIS projects which they need to load terabytes of data in runtime.

In the earlier video I showed here, you see a bigger terrain for the whole area of game world as background mountains with streaming terrains being created on top of it around the player. As you see the concept is simple and you can quickly setup the scene while the results are still satisfactory.

Also there are numerous settings in the “RuntimeOffline” script as the handler for the generating terrains in the scene which you can define the whole world size units which consequently sets each tile size in units

Regarding the floating point issues in a multiplayer environment, this was the main concern of mine too while developing such system and as you recognized it won’t work in that or maybe just makes everything more complicated if implemented. So more R&D in this case is needed or maybe you have to shrink everything to be close to the world center not to get floating issues!

This is the most I could get out of our conversation so far and the real testing whether to select between assets is to have them run side by side which would be so cool to have that. Anyways, let me know of any specific questions regarding your project and I’ll be happy to answer them all.

Hi.
I have changed some parts of the TerrainGenerator.cs script to chose an maximum area of 3000 x 3000km and generate 128 x 128 tiles, but now I have a problem with the low heightmap resolution. I tried to increase it to 131072, but when I click “Generate Heights” it loads for 1 second and then this appears:

at System.Web.Services.Protocols.SoapHttpClientProtocol.ReceiveResponse (System.Net.WebResponse response, System.Web.Services.Protocols.SoapClientMessage message, System.Web.Services.Protocols.SoapExtension[ ] extensions) [0x00000] in <filename unknown>:0
at System.Web.Services.Protocols.SoapHttpClientProtocol.Invoke (System.String method_name, System.Object[ ] parameters) [0x00000] in <filename unknown>:0
at WorldElevation.Terrain_ImageServer.ExportImage (WorldElevation.GeoImageDescription ImageDescription, WorldElevation.ImageType ImageType) [0x00015] in B:\Programme\Unity\Unity Projects\Asset tests\Assets\TerraLand\TerraLand Core\WebServer\App_Code\WorldElevation_MapService.asmx.cs:1769
at (wrapper remoting-invoke-with-check) WorldElevation.Terrain_ImageServer:ExportImage (WorldElevation.GeoImageDescription,WorldElevation.ImageType)
at TerrainGenerator.ElevationDownload () [0x00106] in B:\Programme\Unity\Unity Projects\Asset tests\Assets\TerraLand\TerraLand Core\Editor\TerrainGenerator.cs:5306
UnityEngine.Debug:Log(Object)
TerrainGenerator:ElevationDownload() (at Assets/TerraLand/TerraLand Core/Editor/TerrainGenerator.cs:5311)
<ServerConnectHeightmap>c__AnonStorey0:<>m__0() (at Assets/TerraLand/TerraLand Core/Editor/TerrainGenerator.cs:5192)
TerrainGenerator:RunAction(Object) (at Assets/TerraLand/TerraLand Core/Editor/TerrainGenerator.cs:9384)```

How can I generate a heightmap with this size then? At the moment the maximum is 4096 which generates a 32 heightmap for every terrain, which is very low.

First of all, you just need to study about heightmap resolutions and its limits in a static scene created in editor, the 131072 resolution heightmap you mentioned on a terrain in the scene will probably take a week to render only 1 frame :hushed:
That’s why there is the 4096 limit for the heightmap resolution even though if you try to increase this limit as you did, the server does not allow to obtain such data file with this huge resolution and you will receive the mentioned error.

But hopefully that’s exactly why we have the “Dynamic World” mode in TerraLand to dynamically stream huge datasets with high resolution heightmap/imagery data tiles in it in runtime.

When in Dynamic World mode chosen from the top of the UI, after defining the Area Location & Size, you simply define the total number of tiles by setting the “Tiles Grid” value (in your case 128x128 = 16384 tiles) and then select each heightmap and imagery tile’s resolution. For instance if you want to have the total heightmap resolution of 131072 pixels as in your case, you have to set heightmap tile resolution to 1024 (131072 / 128 = 1024).

The following video tutorial shows you how to setup TerraLand Downloader’s Dynamic World settings to define a large area and obtain its needed elevation/imagery data:

Don’t forget to firstly setup TerraLand Downloader for your Unity installation explained at the beginning of the video.

Also please bear in mind that the data resolution you are trying to obtain still can be huge for most cases even if it is a dynamic streaming system because it will take a lot of time to download data tiles. However if you know the consequences and manage to achieve this, it’s going to be epic as we didn’t test such resolutions ourselves, definitely waiting to see what you will come up with.

But why can I create these terrains with other assets then, but not with this one? I just want 1 1024x1024 heightmap on each terrain. And it doesn’t take a week to render 1 frame. I get my normal 60fps. I just need the terrains with heightmaps on it.
Btw: What is with the ocean? There aren’t any heightmap values for it or? It is then a flat ocen at 0m or? Can you change it somewhere to get a deeper one?

Just reread what you asked for and what I replied, it’s not about this asset or that, no computers on Earth can handle such resolution in a static scene except utilizing a dynamic streaming system such as the one in TerraLand.

Bathymetry data is there but with very low resolution but if you want to exaggerate the depth, the variable lowestPoint is there for this. There are also some discussions for the subject on the TerraLand 2 thread recently and also there are some tips in our discord channel.

Hi,
I have a few questions before buying what is look like an amazing tool (sorry if you already answer some of them, I missed it):
I want to use terraland to display high resolution topographic data for research purposes. I wonder if:

  • Terraland runs under Unity linux version?
  • If I understand well, I can use my own DEM, but can terraland keeps and displays data with resolution lower than 1 meter or will smooth them?
  • How does Terraland will handle nan value in DEM?

Thanks in advance for your answers.

@CornichonCosmique welcome to the community.

We have not tested it ourselves but if Unity is OK on your machine there is no reason that TerraLand won’t work in that!

Yes, you can definitely use your own DEM files and generate terrains out of them. TerraLand does not care about the original pixel resolution (cell spacing) of the heightmap and is capable of generating surfaces from possibly millimeters to kilometers but the total resolution of the file is the factor which is important in a static scene and usually 4k is the max.

So if you have a data with 1 meter pixel resolution and it is 4096x4096, the area would be 4096 m2; if the data has 25 centimeter resolution in a 4k heightmap, it will cover an area of 1024 m2.

All other processors also does not care about the pixel resolution and will operate on top of surface including the Smooth operation.

The nan values existing in standard Ascii Arc Grid (.asc) files are a constant value defined at the file’s header and TerraLand consider them to be the value of 0. However whether the files is an asc or in other format, you can simply define arbitrary value for nan values by a simple flag.

Currently TerraLand won’t automatically fill these gaps and nan values for you but that would be a simple addition to the system which we try to add in future updates. As TerraLand 3 now comes with the source, you can do it yourself or ask us to share that part with you too. Also Global Mapper has an option to fill these gaps for you and normalize empty values from neighbor pixels during the export.

Thanks for the quick answer.

Ok, I will see how to deal with the nan values.
I come from the GIS world and know the different solutions to fill nan values, but here, I need to keep them because it means that we do not have data to interpret/analyze. And because I’m using bathymetric data, the 0 value will be problematic. I will try to automatically set this value to the minimum height of each terrain.
Thanks again

welcome, yes changing nan values to the lowest value is the easiest way if the sea bed shape is not important but changing it corresponding to its neighbor pixels is the best option and is not that hard to implement. Let me know if you need help in this.

Hi,
Just a simply question
Can I export the terrains generated with the downloader in other projects including only the necessary dependencies.
I tried many tests without succeeding.
Can someone help me about this issue?
Thanks
BN

@berry4u That’s one of the main reasons TerraLand is there and the process is so simple.

For this you only need heightmap(s) and satellite image(s) to be present in your project in order to use them in other scenes or projects later.

Regarding heightmaps, you can either

  • Go to “Save Elevation Data” section in TerraLand Downloader’s UI and select which ever format you want to save the heightmap (raw is recommended but if you want to keep the geo-referenced data in file, choose Ascii format)
  • Or use TerraLand Terrain component to insert any terrain(s) in that and use the “Export Elevation Data” processor in the UI to create heightmap(s) out of you terrain surface.

And for the satellite imagery, if “Texture Terrain” option is “On” in the “Satellite Image Downloader” section which is by default, all downloaded imagery files will be saved in your project under TerraLand => TerraLand Core => Downloads in a folder named after the download date. So you can use them for later terrain generation/editing in other projects.

In TerraLand Terrain component there is a processor named “Image Tiler” which can texture inserted terrain(s) using previously downloaded satellite images.

EDIT: TEXTURE NOT SHOWING FIXED. See below original post for the fix

Hello, just bought the asset and set up everything as requested.
I followed the online tutorial

step by step as well…
Everything seems to work, the terrain is generated BUT the texture is not applied.
The texture is correctly downloaded as far as i can tell, it’s just missing from the terrain…
Any advice? Thanks a lot!

EDIT: I found out that force quitting the job when everything is downloaded actually works, and the textures are finally applied - maybe it’s a small bug? I’m on a Mac running Unity 2017.3.
Thanks again

When starting Terraland the first time, i get prompted with two messages:

Is there a specific order i shoud accept settings?
Because they seem to overwrite each other…
And at the moment i’m not really interested in the Tournament stuff…
Thanks

Assets/TerraLand/TerraLand Tournament/Scripts/GameManagerTT.cs(71,13): error CS0246: The type or namespace name `SmoothFollowAdvanced' could not be found. Are you missing an assembly reference?

Nothing else is in the project only this.

All good, removed the tournament.

Thanks for the report, Macs sometimes act weirdly especially in file management and access rights which I think is the case here. Will investigate more on this to find a solution. Does it happen occasionally or every time?

I’d say 99% of the time. It actually worked only once, and it applied textures only to half of the map. The other half was missing textures even if it downloaded all of them :confused:

No, there is no ordering on accepting these launchers. However if for any reason you don’t want them to appear again and change project settings find the 2 scripts named “TerraLandSettings” & “TournamentSettings” and delete them from the project.

But as a reminder, do the one time instructions for each Unity installation explained at the “_IMPORTANT SETUP” folder under TerraLand Core directory in project. The instructions are also explained at the beginning of the following video tutorial:
https://www.youtube.com/watch?v=GmwMbIuFq5U

Regarding Tournament, you can safely exclude them while importing the TerraLand package to only have the core plugin in place or download previously imported files of Tournament from the project.

1 Like

Even weirder! Do you receive any errors or warnings in the console?

As a hint, if all satellite images downloaded successfully you can easily texture generated terrains later using “TerraLand Terrain” component’s “Image Tiler” processor which takes the folder containing image(s) and textures inserted terrain(s) in the scene.