Understanding CopyActiveRenderTextureToHeightmap

[iddqd]

Hi there

I have a question regarding this newer Terrain method: Unity - Scripting API: TerrainData.CopyActiveRenderTextureToHeightmap

Must the GPU be synchronized back to the CPU at all?

If I use TerrainHeightmapSyncControl.None as the third parameter - must I later on synchronize or can I leave it all in the GPU and have a fully working Terrain with Grass etc.?

Also, if I use TerrainHeightmapSyncControl.HeightAndLod - will the readback to CPU be done asynchronously? If not, can I use TerrainHeightmapSyncControl.None and then do the readback myself async?

Hope somebody can help me with this.

Thank you.

[iddqd]

I did some more digging.

I guess one reason for CopyActiveRenderTextureToHeightmap is to get around the slow TerrainData.SetHeights() - however, CopyActiveRenderTextureToHeightmap still seems slow because it doesn’t copy the data back to the CPU asynchonously. If I’m interpreting this correctly, I’m still getting a blocking Gfx.ReadbackImage instead of the newer Unity - Scripting API: AsyncGPUReadback

Perhaps somebody from Unity can elaborate since this is all pretty new? @wyatttt ?

Many thanks!

[wyattt]

heya

@iddqd

Must the GPU be synchronized back to the CPU at all?

depends on what you want to do. if you are doing stuff on the CPU based on the heightmap then you will want to ie if you want edit-time painting or physics to work.

@iddqd

If I use TerrainHeightmapSyncControl.None as the third parameter - must I later on synchronize or can I leave it all in the GPU and have a fully working Terrain with Grass etc.?

if you don’t need to do anything based on the modified terrain (ie cpu collisions) you could probably get by with the sync control set to None. changes will get wiped when serialization kicks in because the heightmap is uploaded again to the GPU

for the grass, i don’t believe it will work correctly. the grass meshes/patches are generated on a CPU thread and reads directly from the CPU-side heightmap.

@iddqd

Also, if I use TerrainHeightmapSyncControl.HeightAndLod - will the readback to CPU be done asynchronously? If not, can I use TerrainHeightmapSyncControl.None and then do the readback myself async?

it will not use async readback and you are getting a block as the texture data is bussed to the CPU. i want to say the reason we dont use the async readback is that those changes made it into editor source after our terrain changes. you can get around this somewhat by doing as you said and handle the syncing yourself and doing it only when you need to

using the async readback might be a little faster but you’d still have to set the heights manually when the request is done

[iddqd]

@wyattt

heya

depends on what you want to do. if you are doing stuff on the CPU based on the heightmap then you will want to ie if you want edit-time painting or physics to work.

if you don’t need to do anything based on the modified terrain (ie cpu collisions) you could probably get by with the sync control set to None. changes will get wiped when serialization kicks in because the heightmap is uploaded again to the GPU

for the grass, i don’t believe it will work correctly. the grass meshes/patches are generated on a CPU thread and reads directly from the CPU-side heightmap.

it will not use async readback and you are getting a block as the texture data is bussed to the CPU. i want to say the reason we dont use the async readback is that those changes made it into editor source after our terrain changes. you can get around this somewhat by doing as you said and handle the syncing yourself and doing it only when you need to

using the async readback might be a little faster but you’d still have to set the heights manually when the request is done

Hey @wyattt

Thanks a lot for replying to this. So in the mean time, I figured out that I will be needing a CPU readback for my case.

Let’s say I do the async readback myself, where must I store the result in TerrainData and trigger the LOD calculations? I can only find SetHeights (or SetHeightsDelayLOD) but this will rebuild the Terrain and is actually the method I’ve been trying to circumvent.

Or if that’s not possible, how about adding TerrainData.SyncHeightmap(bool async) or similar to the API?

It’s not about async being faster (it should be smoother but probably slower), the issue is that SetHeights and now also CopyActiveRenderTextureToHeightmap cause a block resulting in an fps drop which makes procedural/inifinite terrain generation not so smooth as it could be.

Thanks

[wyattt]

You can try doing what this user did and modify the heightmap with NativeArrays and Burst/Jobs:

Requesting example for writing heights to Terrain via heightmapTexture or TerrainPaintUtility

I'm having trouble using the (hopefully faster) API that sets heights on a Terrain object. Some code examples would be really wonderful on how to go about it. No matter what I have tried, the resulting heights just get set far above the maximum height for the Terrain, and are all flat. Here's some example code: Terrain terrainComp = forTerrainComps[tx, ty]; var renderTex = terrainComp.terrainData.heightmapTexture; Texture2D tempTex = new Texture2D(renderTex.width, renderTex.height, UnityEngine…

TerrainData API & NativeArray

Hello, We're doing a lot of procedural generation of terrain data at runtime, and are using the c# jobs system + Burst to make that nice and fast. We essentially generate a bunch of 2D arrays in these jobs, which we then push onto the Terrain with functions like TerrainData.SetHeights. However, these take 2D c# arrays only, and converting between the native arrays and managed arrays has some overhead. I'd like to request versions of the TerrainData.SetXXX functions that take a NativeContainer…

[iddqd]

Hi @wyatttt

Thanks for that, but it seems that he solved using a RenderTexture to set the heightmap without using CopyActiveRenderTextureToHeightmap but I cannot see a solution in those posts to get it from the GPU back to the CPU without the bottleneck. I already have the data in the CPU in my own array, I just want to set it somewhere in the Terrain.

Do you have a solution for this? Is there a way to get a pointer to the CPU Heightmap Data or something similar?

Thank you very much!

[isaac-ashdown]

Hello!

Now we’re on 2019.2.0b7, and I’m using the following (note I’m not using render textures, I’m doing everything using the jobs system, and writing to the texture using Texture2D.GetRawTextureData and writing to that in the jobs. This is primarily because I am not good at shader coding though more than anything else):

// the Vector2Int here is an index into the terrain patch
            List<(HeightCalculationJob, JobHandle, Vector2Int)> calcJobs = new List<(HeightCalculationJob, JobHandle, Vector2Int)>();

            foreach (var heightJob in calcJobs)
            {
                int tx = heightJob.Item3.x;
                int ty = heightJob.Item3.y;
                var dirtyRect = forDirtyRects[tx, ty];

                heightJob.Item2.Complete();

                forHeightTextures[tx, ty].Apply();

                Graphics.CopyTexture(forHeightTextures[tx, ty], forTerrainComps[tx, ty].terrainData.heightmapTexture);

                var dirtySize = dirtyRect.Size;
                forTerrainComps[tx, ty].terrainData.DirtyHeightmapRegion(new RectInt(dirtyRect.min.x, dirtyRect.min.y, dirtySize.x, dirtySize.y), TerrainHeightmapSyncControl.None);

                var job = heightJob.Item1;
                job.GetRemainingDirtyResults(ref forDirtyRects[tx, ty]);

                if (wasDetailDataChanged || !forDirtyRects[tx, ty].isDirty)
                {
                    forTerrainComps[tx, ty].terrainData.SyncHeightmap();
                }
            }

So each job calculates the heightmap that I want to set and puts it on a Texture (forHeightTextures). I then copy that to the terrain component’s heightmapTexture, then mark the region within the heightmap that I was actually working on dirty, passing in None to syncControl. This is all super fast and allows me to interpolate heights over time to create a smooth transition between terrain types - you can see it in action here.

Then, I check if the interpolation has completed, and only then do I call SyncHeightmap. This is the expensive part. It causes the phys geo to be updated, and also the grass(detail) billboards that are on that part of the heightmap get set to the correct height. This causes a pop, which I’d rather avoid. But without it it just runs too slow for our needs.

Today I tried passing in TerrainHeightmapSyncControl.TerrainAndLOD to DirtyHeightmapRegion, and this then updates the phys geo & grass sprites immediately, but is expensive again. I’m not totally sure what the difference betwen doing that and calling SyncHeightmap is, but I guess SyncHeightmap syncs the whole thing, rather than just the dirty region passed in to DirtyHeightmapRegion. But not totally sure - I need to profile it. But just changing that over appears to be still too slow for our game (we have some effects that can cause a number of changes to the heightmap over time, which needs to be buttery smooth. So we’re living with the pops for now.

It would be great though if there was a mode that could be passed in to DirtyHeightmapRegion which only did the necessary calculations for the grass billboards to be correctly positioned every frame, without doing the phys/cpu/LOD update, which I don’t care about so much. Would that be a possibility, @wyatttt ?

It would be nice to have a direct API for the terrain that took NativeArrays, though having this texture-based API instead is not so bad, as the texture API itself can take NativeArrays. And the bottleneck appears to be the sync step anyway.

[iddqd]

Hi @isaac-ashdown

Yes that’s where I am now, living with the pop. The pop comes from when the GPU texture is read back to the CPU, that’s the bottleneck. I’m pretty sure that the lag will always be the same, whether you do it immediately with TerrainHeightmapSyncControl.TerrainAndLOD or TerrainHeightmapSyncControl.Non and then sync later on. SetHeights has pretty much the same performance for me now again.

Unity does have a method to read RenderTextures Asynchronously back to CPU which would probably remove the pop: Unity - Scripting API: AsyncGPUReadback

But it’s not implemented for the Terrain readback. Keeping my fingers crossed for a solution.

[iddqd]

Hi @isaac-ashdown

I’ve had some good results by using a Coroutine to apply the heightmap over several frames.

So I’m using a 1024 Heightmap on a Terrain, but I’m splitting it up into several chunks (4 or 16 works good for my case) and applying them partially to the terrain. Performance is much better, you should try it.

I’m using CopyActiveRenderTextureToHeightmap - I will try the same approach with SetHeights perhaps later on.

Edit: I see I could probably also use DirtyHeightmapRegion for this.

Good luck

[iddqd]

Hi @wyatttt

Are there perhaps any plans to make the CopyActiveRenderTextureToHeightmap readback optionally Async?

Splitting CopyActiveRenderTextureToHeightmap into several calls helps a lot, but having it Async would most likely be the most performant solution.

Thank you

[isaac-ashdown]

Hi @iddqd - which call is it exactly you’re spliiting up over several frames? Doesn’t that lead to a noticable effect whereby parts of the heightmap aren’t lined up with each other?

Would you mind posting your coroutine code?

[thomas-weltenbauer]

Yes it would be interesting to see your coroutine code, @iddqd !
Maybe you could share it?

[iddqd]

So let’s say I apply a 1024 Heightmap to the Terrain, I now do this as follows using CopyActiveRenderTextureToHeightmap.

(rtHeight is the Heightmap in a Rendertexture)

public static IEnumerator RenderTextureToTerrainChunks(Terrain currentTerrain, RenderTexture rtHeight, int chunks, float secondsForFullHeighmap)
    {
        int numChunksPerRow = (int)Mathf.Sqrt(chunks);
        int chunkResolution = 1024 / numChunksPerRow;
        float waitTime = secondsForFullHeighmap/ chunks;

        for (int x = 0; x < numChunksPerRow; x ++)
        {
            for (int y = 0; y < numChunksPerRow; y++)
            {
               
                yield return new WaitForSecondsRealtime(waitTime);

                int offset = 64;
                RectInt rect = new RectInt(offset + (chunkResolution * x), offset + (chunkResolution * y), chunkResolution, chunkResolution);
                Vector2Int vect = new Vector2Int(chunkResolution * x, chunkResolution * y);
               
                if(x == numChunksPerRow -1)
                {
                    rect.width += 1; // make it 1025
                }

                if(y == numChunksPerRow -1)
                {
                    rect.height += 1; // make it 1025
                }

                RenderTexture rtActiveOld = RenderTexture.active;
                RenderTexture.active = rtHeight;
                currentTerrain.terrainData.CopyActiveRenderTextureToHeightmap(rect, vect, TerrainHeightmapSyncControl.HeightAndLod);
                RenderTexture.active = rtActiveOld;
            }
        }
    }

This will split the 1024 HM into let’s say 16 chunks and apply these 16 over a duration of secondsForFullHeighmap. So yes, there will be times when the full Terrain is not displayed correctly with incorrect seams, you will maybe want to hide this somehow (add the terrain a few km ahead of the player perhaps).

Also what I think helped performance was to use Unity - Scripting API: TerrainData.CopyActiveRenderTextureToTexture to apply the Alpha/Splat maps.

[iddqd]

Of course, I think it would be best if Unity added something like this:
TerrainHeightmapSyncControl.AsyncHeightOnly
TerrainHeightmapSyncControl.AsyncHeightAndLod

[crysicle]

@iddqd

Of course, I think it would be best if Unity added something like this:
TerrainHeightmapSyncControl.AsyncHeightOnly
TerrainHeightmapSyncControl.AsyncHeightAndLod

I 've opened up a discussion about the lack of this option a few months ago here . Would be really great if this was actually implemented.

I’ve been trying to come up with a way to have efficient manipulation of the heightmap for months now with all of the LOD and physics synced properly. These are my findings:

• If you want scarce heightmap manipulation, about 1-3k heightmaps in total each frame, use the job system to calculate the data and update it through SetHeights() or SetHeightsDelayLOD(), otherwise the overhead will greatly stall the CPU. Doing it this way will sync the LOD and the collision data without the need to create your own workarounds/systems to sync everything.

@wyattt do you perhaps know if any NativeArray support is coming to the SetHeights() method similar to what is happening in this link to meshes?

-If you want to update 3k - 4kk or more heightmaps each frame without lag, use the CopyActiveRenderTextureToHeightmap() and do all heightmap manipulation via textures. Currently there is no way to actually sync heightmaps or LOD without huge stalls with Unity’s API each frame. Though i’ve come up with a few workarounds for this using AsyncGPUReadback.

1. To calculate detail positions/rotations, do all of the calculations inside a shader relative to the texture values rather than the data on the CPU. Gather detail positions relative to the heightmap, send them to a shader, calculate new positions relative to the texture and retrieve the data with AsyncGPUReadback and then assign it back manually.

2 To sync terrain colliders for physics, split the terrain collider into multiple meshes and retrieve the data using AsyncGPUReadback. I can retrieve about 1kk heightmaps each frame from the GPU no problem with this. However, after assigning the vertices to the mesh, there are baking problems with physX which really only allow about 2k-3k vertice manipulation on mesh colliders each frame before severe lag kicks in.

@wyattt do you perhaps know why physX needs to rebake CollisionData for a mesh after assigning new vertices, but not for the terrain? After using SyncHeightmaps() i’ve noticed that the lag only comes from GPU stall and an internal RecomputeInvalidPatches() function. I created a thread here , though haven’t gotten any response yet.

3. Vert/Tris LOD syncing isn’t possible through AsyncGPUReadback due to Unity’s black box problem discussed here .

4. If you need the manipulated heightmap visual effects to be synchronized with physics you’ll also need an additional heightmap for every other heightmap. Once you start deforming the heightmap, update the first heightmap instead, use it as a data source for asynchronous detail position/rotation and mesh creation. Once the physics data is retrieved and applied, update the second heightmap, which would be the actual terrain heightmap.

If the Async functions were to be introduced, it’d also be great if an additional source besides the terrain’s heightmap could be used for LOD and terrain collider data, such as an additional heightmap to which we could write first. This would allow to add an effect to the heightmap which is invisible, promt an async request and update all of the data in one go seamlessly : visible heightmap, terrain collider, detail positions/rotations, and vert/tris LOD.

[iddqd]

@crysicle

I 've opened up a discussion about the lack of this option a few months ago here . Would be really great if this was actually implemented.

I’ve been trying to come up with a way to have efficient manipulation of the heightmap for months now with all of the LOD and physics synced properly. These are my findings:

• If you want scarce heightmap manipulation, about 1-3k heightmaps in total each frame, use the job system to calculate the data and update it through SetHeights() or SetHeightsDelayLOD(), otherwise the overhead will greatly stall the CPU. Doing it this way will sync the LOD and the collision data without the need to create your own workarounds/systems to sync everything.

@wyattt do you perhaps know if any NativeArray support is coming to the SetHeights() method similar to what is happening in this link to meshes?

-If you want to update 3k - 4kk or more heightmaps each frame without lag, use the CopyActiveRenderTextureToHeightmap() and do all heightmap manipulation via textures. Currently there is no way to actually sync heightmaps or LOD without huge stalls with Unity’s API each frame. Though i’ve come up with a few workarounds for this using AsyncGPUReadback.

1. To calculate detail positions/rotations, do all of the calculations inside a shader relative to the texture values rather than the data on the CPU. Gather detail positions relative to the heightmap, send them to a shader, calculate new positions relative to the texture and retrieve the data with AsyncGPUReadback and then assign it back manually.

2 To sync terrain colliders for physics, split the terrain collider into multiple meshes and retrieve the data using AsyncGPUReadback. I can retrieve about 1kk heightmaps each frame from the GPU no problem with this. However, after assigning the vertices to the mesh, there are baking problems with physX which really only allow about 2k-3k vertice manipulation on mesh colliders each frame before severe lag kicks in.

@wyattt do you perhaps know why physX needs to rebake CollisionData for a mesh after assigning new vertices, but not for the terrain? After using SyncHeightmaps() i’ve noticed that the lag only comes from GPU stall and an internal RecomputeInvalidPatches() function. I created a thread here , though haven’t gotten any response yet.

3. Vert/Tris LOD syncing isn’t possible through AsyncGPUReadback due to Unity’s black box problem discussed here .

4. If you need the manipulated heightmap visual effects to be synchronized with physics you’ll also need an additional heightmap for every other heightmap. Once you start deforming the heightmap, update the first heightmap instead, use it as a data source for asynchronous detail position/rotation and mesh creation. Once the physics data is retrieved and applied, update the second heightmap, which would be the actual terrain heightmap.

If the Async functions were to be introduced, it’d also be great if an additional source besides the terrain’s heightmap could be used for LOD and terrain collider data, such as an additional heightmap to which we could write first. This would allow to add an effect to the heightmap which is invisible, promt an async request and update all of the data in one go seamlessly : visible heightmap, terrain collider, detail positions/rotations, and vert/tris LOD.

Interesting stuff, thanks for that input.

[gilley033]

I have a weird issue with an Asset Store product that is using DirtyHeightmapRegion. I’ve tested with varying sizes for the region that is dirtied, all the way from a 65 x 65 region down to a 5 x 5 region of the heightmap, and no matter the size the performance of the method is the same (~54ms on built Player). I’d post the code but it’s obviously not mine to post. Any ideas on what could be happening here?

[Skotrap7]

A bit of an old thread, but I have a heightmap as a render texture and when I use this function it appears it is incorrectly setting the heights and resulting in some weird clipping issues. It seems that this method is creating terrains with 0.5 as the maximum height value, not 1, whereas setting heights from an float[,] uses 1 as the maximum height.

On 2019.4.5f1
Terrain size: (1024, 1024, 1024)
Heightmap Resolution: 513
Heightmap full of values of 0.5

The terrain is a plane at point 1024, whereas I would expect it to be at 512.

Am I forgetting to do something here?

[wyattt]

@Skotrap7

The terrain is a plane at point 1024, whereas I would expect it to be at 512.

Could you explain this bit for me? Are you talking about the y-position of a flattened terrain height at .5 for the heightmap values?

@Skotrap7

A bit of an old thread, but I have a heightmap as a render texture and when I use this function it appears it is incorrectly setting the heights and resulting in some weird clipping issues. It seems that this method is creating terrains with 0.5 as the maximum height value, not 1, whereas setting heights from an float[,] uses 1 as the maximum height.

This is correct. The heightmap is actually a signed texture but we only use half of the range atm. Which is why on the GPU, values are clamped between 0 and .5. On CPU, it might be 0 - 1 and expects -1 - 0 for the other half of the range but I honestly haven’t tried that so I am just guessing for that part and I don’t even know what passing non-unsigned normalized values to SetHeights would do.

[Skotrap7]

@wyattt

Could you explain this bit for me? Are you talking about the y-position of a flattened terrain height at .5 for the heightmap values?

The values in the heightmap are at 0.5 which maxes the terrain height. Values above 0.5 “work” but cause very weird clipping issues since it sounds like they are out of range…

@wyattt

This is correct. The heightmap is actually a signed texture but we only use half of the range atm. Which is why on the GPU, values are clamped between 0 and .5. On CPU, it might be 0 - 1 and expects -1 - 0 for the other half of the range but I honestly haven’t tried that so I am just guessing for that part and I don’t even know what passing non-unsigned normalized values to SetHeights would do.

Sounds like it is working as designed then. Would be cool if the docs specified that the value ranges are different.

Thanks for the quick response, now that I know to shift my heightmap scale everything is pretty awesome.