How do I correctly sample a render texture for tex2Dlod in vertex shader?

[Buttermilch]

Using Unity 2020.3 and standard render pipeline.

I’m using GPU instancing for grass and I want to push it by the player position. For this I’m recording the players position with a extra camera und rendering the result to a rendertexture.

The shader now reads that render texture with tex2Dlod and should deform the vertices.

But in my case the result from tex2Dlod is always 0 and I don’t know why.

Here is my shader setup:

struct v2f {
float2 uv : TEXCOORD0;
UNITY_FOG_COORDS(1)
float4 vertex : SV_POSITION;          
};

sampler2D _GrassCaptureTex;
float4 _GrassCaptureTex_ST;
StructuredBuffer<float4x4> trsBuffer;//transformation matrices

v2f vert (appdata v, uint instanceID : SV_InstanceID) {
v2f o;

float3 positionWorldSpace = mul(trsBuffer[instanceID], float4(v.vertex.xyz, 1));
float4 renderTexValue = tex2Dlod(_GrassCaptureTex, float4(v.uv, 0, 0));
positionWorldSpace.x += renderTexValue.r * 6;

o.vertex = mul(UNITY_MATRIX_VP, float4(positionWorldSpace, 1));
return o;
}

[burningmime]

The uv you’re using is the uv of the vertex of the mesh. So it would be like (0,0) at the bottom of the grass mesh and (0,1) at the top of the grass mesh or something. I assume you actually want to sample the render texture from the screen position where the grass will be.

[Buttermilch]

@burningmime

The uv you’re using is the uv of the vertex of the mesh. So it would be like (0,0) at the bottom of the grass mesh and (0,1) at the top of the grass mesh or something. I assume you actually want to sample the render texture from the screen position where the grass will be.

How would I do that?
I tried it with the world position of the vertex and convert that into screen position but still does not really work.
Here’s my attempt:

    float4 worldToScreen = UnityObjectToClipPos(v.vertex);
                float4 screenPos = ComputeScreenPos(worldToScreen);
                float4 renderTexResult =tex2Dlod(_GrassCaptureTex, screenPos);
                positionWorldSpace.x += renderTexResult.x;

[AcidArrow]

To rule out something going wrong with the rendertexture, if you plug a random texture that has stuff in the red channel for _GrassCaptureTex does any deformation happen?

[Buttermilch]

@AcidArrow

To rule out something going wrong with the rendertexture, if you plug a random texture that has stuff in the red channel for _GrassCaptureTex does any deformation happen?

Yes something happens but it doesn’t look right (looks like all vertices from all instances are moved at once). I think it has to do with the sampling.

[mgear]

try apply that sampled color into vertex color, and use shader that displays vertex colors,
to see what colors they receive from texture.
(test with some debug texture that has different colors on each corner or so)

[Neto_Kokku]

How is _GrassCaptureTex generated and what it contains? You need to calculate what UV coordinates each grass vertex will read from based on what coordinates you are using to write into the grass texture.

[Buttermilch]

@Neto_Kokku I’m generating the render texture from these trails:

And the render texture looks like this with a resolution of 1024 x 1024:

@mgear I’ve send over my UVs to the fragment shader and with my first approach from my post I can see that the texture is sampled in object space (?)

[Neto_Kokku]

You’ll need to pass the information about the world-space rectangle used to render into the grass texture to the grass shader, use that to calculate the position of the grass instance inside that rectangle in the 0-1 range to use as UV to sample the texture.

[Buttermilch]

@Neto_Kokku

You’ll need to pass the information about the world-space rectangle used to render into the grass texture to the grass shader, use that to calculate the position of the grass instance inside that rectangle in the 0-1 range to use as UV to sample the texture.

Sounds good but I can’t figure out how to do that exactly.
I thought about using this for my UVs (1024 is the resolution of the texture) but this gives me weird results:

  float3 posRelativeToCam = normalize(_cameraPosition - positionWorldSpace) * 1024;

[Neto_Kokku]

Research about how to project a texture in a shader, because that’s basically the same thing.

[burningmime]

@Buttermilch

Sounds good but I can’t figure out how to do that exactly.
I thought about using this for my UVs (1024 is the resolution of the texture) but this gives me weird results:

  float3 posRelativeToCam = normalize(_cameraPosition - positionWorldSpace) * 1024;

UVs range from 0 to 1. You should not multiply by the texture size.

[Buttermilch]

I’ve found the solution in this video

Unity | Interactive Snow Shader | Stylized Setup URP + Built-In

Basically I needed the relative position of the vertex to the camera (only x and z components).
Then using this as the UV and dividng it by 2 times the size of the orthographic camera. At last add 0.5 to center it and it works now.

 float3 worldPos = mul(trsBuffer[instanceID], float4(v.vertex.xyz, 1));
                float2 uv = worldPos.xz - _CameraPosition.xz;
                uv = uv / (50 * 2); //_OrthographicCamSize
                uv += 0.5;
                float4 renderTexValue = tex2Dlod(_RenderTex, float4(uv, 0, 0));